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The Principal Pipeline Initiative In Action 


Preface 


This is the fifth report from an evaluation of the Principal Pipeline Initiative, in which six large urban 
school districts have received support for strengthening novice principals’ capabilities through specific 
strategies. The Wallace Foundation is sponsor of the initiative and of the evaluation, which is conducted by 
Policy Studies Associates, Inc., and RAND. This report analyzes implementation of the initiative in the par- 
ticipating school districts from grant award in August 2011 through spring 2015. This study of the initia- 
tive’s implementation was designed to draw lessons about the process and early results of the work done in 
districts with varying histories in different states and regions. It emphasizes approaches that the districts 
took and results that emerged during almost four years of implementation. Intended audiences for the re- 
port include other school districts that might embark on similar approaches and policymakers who might 
support such efforts. 
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Executive Summary 


S trong principals are central to improving schools — indeed, leadership is second only to teaching 
among school-related factors that influence student achievement. Districts struggle, however, to de- 
velop a sufficient pool of highly capable principals. While research has identified strategies that are ef- 
fective in preparing and supporting school leaders, few districts have pulled together a coherent set of 
strategies to form a pipeline to the principalship. 

Recognizing the need to improve the supply of high-quality principals, The Wallace Foundation launched 
the Principal Pipeline Initiative (PPI) in 2011. The goal was to test the proposition that districts could pro- 
duce a large cadre of strong novice principals by making a concerted effort to implement a set of interre- 
lated policies and practices, and that doing so would positively affect school outcomes. Participating 
districts focused on implementing four key components: 


Adopting standards of practice and performance to guide principal preparation, hiring, evaluation, 
and support 

Improving the quality of preservice preparation for principals 
Using selective hiring and placement practices to 
match principal candidates with schools 
Implementing on-the-job evaluation and support 
for novice principals (those in their first three 
years on the job) 

Six large districts received multi-year grants of $8.5 mil- 
lion to $13.25 million from the foundation to cover a 
portion of the start-up costs of developing a principal 
pipeline. The districts were selected partly because they 
had already adopted some policies and practices consis- 
tent with the PPI components. The foundation also pro- 
vided technical assistance. The PPI sought not only to 
institute, grow, and sustain the key components within 
each participating district, but also to generate examples 
that other districts might follow. 


Principal Pipeline Initiative 
Districts 

Charlotte-Mecklenburg Schools, North 
Carolina 

Denver Public Schools, Colorado 
Gwinnett County Public Schools, Georgia 
Hillsborough County Public Schools, 
Florida 

New York City Department of Education, 
New York 

Prince George's County Public Schools, 
Maryland 



This report is the fifth and last in a series on PPI implementation that is part of an evaluation by Policy 
Studies Associates and RAND. Data for the evaluation were collected from late 2011 through spring 2015, 
using interviews with district administrators, surveys and focus groups of novice principals, and reviews of 
district documents. Details about study methods appear in the body of this report. 


This culminating report summarizes findings about implementation of the four PPI components across the 
six districts, as of spring 2015 when all components were expected to be up and running. The report details 
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the implementation approaches, accomplishments, and challenges of participating districts; identifies fac- 
tors that helped or impeded their progress; highlights lessons learned; and presents implications for policy- 
makers and other districts. An evaluation report about the effects of the PPI on student achievement is 
scheduled for publication in 2018. 

This section summarizes — 

An overall finding about implementation across all six districts. 

Main findings about the implementation of each PPI component. 

Factors contributing to progress in PPI districts. 

Early evidence of results from the perspectives of novice principals and district leaders. 

Persistent challenges faced by districts in redesigning principal pipeline systems. 

Implications for other school districts and policymakers. 


Overall Implementation 

To a striking extent, all six districts carried out the kinds of policies and practices called for by 
the PPI. 



All of the districts marshalled the will and capacity to put in place the key PPI components by spring 2015. 
Consistent with PPI guidance, these districts built tools, programs, partnerships, data systems, and staff ca- 
pabilities for supporting novice principals. 


Local commitment to the initiative actually appeared to strengthen over the years. Districts maintained a 
focus on the key PPI components despite superintendent transitions in four districts and the many other ini- 
tiatives under way in all of the districts. District leaders and project directors approached their tasks not as 
hoops to jump through for a funder but as part of a coherent design. 


Standards 


Defining standards and competencies for principals was a powerful first step toward creating a 
solid principal pipeline. 


Districts actively used standards to help align their principal preparation programs, hiring criteria, and on- 
the-job evaluation rubrics with district priorities for school leadership. The standards brought coherence to 
district actions and provided a common language for discussing school leadership. 


Standards-writing was not a one-time undertaking but an iterative process. 


All districts revised their standards to meet practical challenges as other parts of the pipeline evolved. For 
example, they clarified wording when questions arose in principal evaluations. 


Principal Preparation 

Districts made progress in using data to identify potential principals, creating or strengthening 
their own preservice programs, and collaborating with universities. 


All six districts started or expanded their own district-run principal preparation programs for high-poten- 
tial assistant principals. Moreover, all districts worked in partnership with university programs to prepare 
aspiring leaders. As part of these preservice efforts, all districts encouraged principals and other district 
staff to find and cultivate rising talent. 

District leaders and their partners emphasized the value of gathering and systematically using data to con- 
tinually assess their leader preparation efforts. With training provided by the PPI, districts used a suite of 
tools (Quality Measures for Education Leadership Systems and Programs) that included indicators and 
rubrics reviewing program effectiveness in the areas of course content and pedagogy, clinical practice, re- 
cruitment and selection, and performance of program graduates. 

While district leaders also provided on-the-job leadership experiences as part of preservice preparation for 
principals, they struggled to mobilize enough sites and mentors. 


Principal Hiring and Placement 

Districts acted quickly to make hiring, placement, and principal succession planning more 
systematic and informed by data. 


At the start of the PPI, leaders in all participating districts had at least some concerns about their districts’ 
hiring and placement procedures. These leaders were uncertain whether they were placing the best possible 
candidate in each school. Most reported using an unsystematic process to fill vacancies one at a time and 
bemoaned the lack of hard, usable data about candidates. 

Districts took early steps to align their hiring procedures and candidate assessments with district standards and 
competencies for principals. Over the next few years, districts made several improvements in screening and 
placing principals. By 2014, all districts were using multiple stages of scrutiny and assessment to develop a future 
pool of high-quality candidates to fill principal vacancies as they arose. By 2015, all districts required aspiring 
candidates to engage in practical demonstrations of their skills, typically with simulated scenarios. 
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A crucial step in improving hiring, placement, and succession planning was the development and use of lon- 
gitudinal data systems, called Leader Tracking Systems (LTS), that collected and maintained information on 
individual candidates’ experience, performance, and assessed competencies. While districts had already 
begun drawing on data in the hiring process, the LTS made this process easier by pulling together a variety 
of data in a more usable form. 


Evaluation and Support for Novice Principals 

Districts implemented standards-based systems for evaluating principals' professional practice. 


During the course of the PPI, every district designed or adapted evaluation rubrics for supervisors to use to 
rate principals on the district’s standards and competencies for school leaders. These evaluations covered 
such areas as instructional leadership, human resource management, school climate, planning, and school 
management and operations. 

In all PPI districts, evaluations became less like a once-a-year meeting and more like structured conversa- 
tions between principals and their supervisors. Comments from focus groups suggested that evaluations 
often involved frequent and intensive interactions between principals and supervisors and ongoing discus- 
sions of principals’ progress toward goals, 
iv 

Most novice principals surveyed viewed these evaluation systems as fair and constructive. 


Novice principals received critical feedback from these evaluation systems. More than half of new princi- 
pals were told after their 2013-14 evaluations that they needed to improve at least one area of practice, 
most often related to instructional leadership. Still, survey responses showed that principals had positive 
perceptions of their district’s evaluation process. Challenges remained in making evaluation rubrics more 
specific and in balancing a desire to differentiate based on principals’ experience or school context against 
the desire to maintain uniformly high expectations. 


The roles of principal supervisors shifted, with smaller caseloads and a greater focus on instruc- 
tional leadership. 


PPI districts invested their own resources to hire more principal supervisors so that each could work with 
fewer principals. Districts also redefined the role of supervisors so they spent less time overseeing compli- 
ance and operations and more time helping principals succeed as instructional leaders. The Wallace 
Foundation grants supported training of the new supervisors. 

Supervisors took on pivotal roles in providing on-the-job evaluation and support for principals. A large 
majority of new principals valued the support received from supervisors. 


Principal supervisors were still adapting to their shifting roles, however, and their level of coaching skills 
varied. Moreover, some supervisors were still expected to carry out administrative tasks, leaving less time 
to provide support. Districts continued to work on developing supervisors’ capacity. 

Mentors and coaches stood out as the most highly valued source of support among the new principals they 
worked with, according to surveys and focus groups. Day-to-day, hands-on support from a mentor or 
coach helped new principals navigate the challenges of their demanding job. 


District leaders were unsure how — or whether — to continue to use group professional develop- 
ment for on-the-job learning. 


In addition to providing induction programs for novice principals and other PPI-related supports, districts 
continued to offer an array of traditional professional development courses and workshops for principals. 
In focus groups, principals described these group opportunities as “compliance-oriented,” intended for 
mass consumption, and overloaded with information; and in a survey rated them less highly than they did 
one-on-one support from mentors, coaches, and supervisors. 


Districts tried to find ways to customize their professional development offerings to align them better with 
principals’ needs and their plans for professional growth. But building a professional development system 
that responds directly, practically, and quickly to principals’ needs is challenging and complex work. As of 
2015, no district claimed to have successfully done it. The relatively low ratings for traditional district-run 
professional development raise a more fundamental set of questions about whether it is worthwhile keeping 
and reshaping these types of courses and workshops. 



Rethinking the Position of Assistant Principal 

PPI districts took steps to systematically select, induct, and coach assistant principals to 
strengthen the pathway to a principalship. Challenges remained, however, in reconciling the 
instructional leadership and managerial expectations of the assistant principal's position. 


In each of the PPI districts, more than two-thirds of novice principals responding to the survey — and some- 
times more than 90 percent — came from the ranks of assistant principals. The experience gained in the 
position was a valuable form of learning — more important than university preparation according to survey 
responses. 

To strengthen the role of assistant principalship in the leadership pipeline, PPI districts sought to use the 
position as both an apprenticeship and a proving ground for future principals. Most of the PPI districts 
built standards and evaluation measures for assistant principals that were based on those for principals. 
Districts also mounted training programs, in-house or with universities, to prepare assistant principals to 
become instructional leaders in their schools. Some districts provided coaching or mentoring for novice 
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assistant principals beyond what they would normally get from their principal, or offered other types of 
induction programs. 

Still, assistant principals continued to spend much of their time on managerial duties, which limited their 
opportunities to practice instructional leadership on the job. 

Districts also had to work with the many assistant principals who would not become principals. 


All of the districts had more assistant principals than principal slots, and not all assistant principals showed 
high potential to advance to a principalship. Thus, districts still faced the challenge of moving some of their 
assistant principals off the track to a hoped-for promotion. 

While these assistant principals could still fill districts’ needs for career managers to help run their schools, 
districts struggled to find ways to differentiate the instructional leadership and managerial functions of the 
assistant principal’s job. Some district leaders wondered whether they should restructure the position. None 
claimed to have good solutions to this dilemma. 

Factors Contributing to Progress 



Three features of the PPI design — local latitude, engagement of top district leaders, and regu- 
lar opportunities for project directors to learn and share — were particularly helpful to districts. 


Observations and interviews suggested that three design features were important to districts’ progress in 
strengthening their principal pipelines. First, although the PPI components were well-defined, districts had 
the latitude to use different strategies to carry them out and shift their emphases over time. In this way, dis- 
tricts took charge of their own work. 

Second, the initiative was designed to engage top district leaders in matters that might otherwise have been 
handled by human capital departments. Superintendents had ongoing conversations with the foundation 
and attended the PPI professional learning community meetings. This involvement of top leaders appeared 
to sustain high-level support for the initiative within districts, cross departmental lines, and help overcome 
bureaucratic resistance. 

Third, the project directors from the six districts met regularly to share ideas and spend uninterrupted dis- 
cussion and planning time with key members of their own local leadership team. They also had regular op- 
portunities to meet with outside experts and visit each other’s districts to gain deeper knowledge of the 
initiative. The project directors said their work benefited from these opportunities for interaction. 


Districts viewed their principal pipelines as a work in progress and continued to fine-tune them. 


Most new tools, programs, and procedures were introduced as pilots. This helped ease the stress about is- 
sues like principal evaluation and enabled districts to spot and correct unanticipated problems. Even after 
these systems became more well-established, districts worked on each pipeline component every year. No 
district leader characterized any part of the pipeline design as completed and unchangeable. Instead, they 
continued to refine their systems to incorporate new knowledge, fix flaws, and address new issues. 


Evidence of Results 

Since it takes years for changes in principal preparation to affect a new crop of principals, it is 
too early to expect to see the full effects of the PPI on incoming principals. 


The pathway to a principalship usually involves university preparation and a stint as an assistant principal, 
and often includes another district-sponsored training program. In the PPI districts, the median time 
elapsed from the beginning of formal preparation to the first principal’s job ranged from three to ten years, 
according to reports from first- and second-year principals in 2015. 

Most of the novice principals surveyed had started their formal preparation before their district had begun 
to implement PPI interventions. Thus, the PPI had not fully reshaped their preparation experience. As of 
spring 2015, only a minority of incoming principals in PPI districts had obtained all of their preparation 
through new or revised programs that reflected the PPI reforms. 


Early evidence shows some positive results of changes made by PPI districts in principal preparation, 
hiring, placement, and evaluation. 

PREPARATION. The most notable change during the evaluation period was a statistically significant in- 
crease in the percentage of novice principals who reported on surveys that their preparation had given them 
“a strong orientation to the principalship as a career.” 

HIRING AND PLACEMENT. Districts seemed to get better at placing principals where they were needed. 
Over the course of the PPI, a higher percentage of novice principals reported that their skills were an “ex- 
cellent” fit for their school’s needs. Top district leaders also viewed changes in hiring and placement as suc- 
cessful. In interviews conducted in 2014 and 2015, most leaders said that incoming principals and assistant 
principals were demonstrating strong leadership capabilities, especially in instructional leadership. 
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EVALUATION AND SUPPORT. New principals continued to give good ratings to their mentoring or 
coaching assistance and gave higher ratings over time to principal supervisors. And despite the introduction 
of new evaluation practices and measures, new principals’ ratings of district evaluation systems were posi- 
tive and held steady. 


Persistent Challenges 

Districts faced persistent challenges in implementing redesigned principal pipelines. 


As of 2015, the PPI districts still had work to do to develop the capacity of principal supervisors and other 
staff to carry out their redesigned pipeline systems. For example, principal supervisors were continuing to 
learn how to evaluate and coach new principals, even as their time was consumed by administrative tasks 
in some cases. 

In addition, district leaders reported mixed effectiveness in reshaping university preparation programs over 
which they had little influence. 

For some stubborn challenges, research and practical wisdom offered no clear solutions. For example, dis- 
tricts remained unsatisfied with the assessments available to screen aspiring principals for the interpersonal 
skills needed in organizational leadership. They struggled to tailor conventional group professional devel- 
opment to individual needs, and to balance tensions in the assistant principal’s role. 


Implications for Districts 

The experience of the PPI districts offers lessons for other districts interested in doing similar 
work, even districts that are smaller or lack the level of support found in PPI districts. 


The following steps would be feasible and could be productive in other large districts, and they could be 
adapted for districts of any size: 

Defining standards and competencies for principals, or drawing on the recently released Professional 
Standards for Educational Leaders, is a powerful first step with several potential benefits. 

Procedural changes in the hiring process, such as requiring candidates to demonstrate their skills 
and drawing on consultants to help identify performance tasks, is a good starting point for improv- 
ing the caliber of principal candidates. 



District-run preparation programs that are specifically tailored to district priorities and needs and 
that give hiring managers a chance to observe aspiring principals can be advantageous. 

Leader Tracking Systems — or, for small districts, less elaborate records of individual leaders’ ac- 
complishments and career paths — offer useful information for hiring and succession planning. 
Eventually the results can be used to improve all parts of the pipeline. 

Introducing changes to pipeline components on a pilot basis allows opportunities to learn from ex- 
perience and revise the components. This can also ease concerns about potential negative impacts. 


Policy Implications 

Building on the experience of this initiative, policymakers could help districts develop stronger 
corps of novice principals to lead school improvement. 


Examples of supportive policies include the following: 


Supporters of reform could ensure that districts have access to the Professional Standards for 
Education Leaders and examples of job descriptions, evaluation rubrics, and other usable materials 
based on these standards. 



Agencies that oversee or support work on principal pipelines could consider a mix of tightly de- 
fined components and local latitude for invention in implementation similar to the PPL 


Seed money could provide expert technical help with hiring procedures or data systems, initial 
training for principal supervisors and others taking on new roles, opportunities to visit and learn 
from other districts, and other important start-up resources. 


More work needs to be done to expand the knowledge base and develop and systematically test 
new approaches for creating a pool of highly capable novice principals. 


Policymakers should also recognize that implementing the components of a major, system-level initiative is 
a multi-year endeavor and that it takes even longer to see the effects of those changes. Patience is required. 
Still, the PPI districts have demonstrated how to take promising steps in that direction while striving for 
continuous improvement. 
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1. Introduction 


T he Wallace Foundation launched the Principal Pipeline Initiative in 2011 because evidence had 

shown that school principals are second only to teachers among school-related factors that affect 
student achievement (Seashore Louis, Leithwood, Wahlstrom, et ah, 2010). The initiative aimed to 
improve the capacity of newly appointed principals in a set of large urban school districts through improve- 
ments in preparation, hiring, evaluation, and support, potentially generating an example for other districts 
to follow. This report is part of an evaluation of the Principal Pipeline Initiative. It summarizes findings of a 
four-year study of the ways in which the districts carried out the initiative, highlighting what they accom- 
plished, what they learned, and how other districts could apply these lessons. 

Six urban school districts participated in the initiative: : 

Charlotte-Mecklenburg Schools, North Carolina 

Denver Public Schools, Colorado 

Gwinnett County Public Schools, Georgia 

Hillsborough County Public Schools, Florida 

New York City Department of Education, New York 

Prince George’s County Public Schools, Maryland 

The evaluation team studied the work that district leaders did from 2011 through spring 2015, which was 
the period in which they were expected to put in place their major strategies for improving new principals’ 
instructional leadership. The districts made concerted efforts to strengthen the preparation and support of 
new principals in a systemic fashion. They addressed four interrelated components of policy and practice, 
identified by The Wallace Foundation (and described in more detail in the next section of this introduction): 

1. Leader standards and aligned job descriptions, preparation, selection, evaluation, and support. 

2. Preservice preparation that includes selective admissions to high-quality programs. 

3. Selective hiring, and placement based on a match between the candidate and the school. 

4. On-the-job evaluation and support addressing the capacity to improve teaching and learning, with 
support focused on needs identified by evaluation. 
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The districts were selected in part because they had already launched some policies and practices consistent 
with these components. The foundation provided multi-year support that included grants of $8.5 million to 
$13.25 million over five years, 1 along with technical assistance, to support the districts in expanding on ex- 
isting practices and building strong, sustainable policies and investments in school leadership. 

The Principal Pipeline Initiative was designed to test the proposition that a combination of programs, part- 
nerships, policies, and practices could result in cohorts of strong novice principals who would make a dif- 
ference in school outcomes. The foundation’s 2011 request for proposals from districts expressed the 
initiative’s central hypothesis in the following terms (Wallace Foundation, p. 2): 


If an urban district and its principal training programs provide large numbers of talented, aspiring 
principals with the right “pre-service ” training and on-the-job evaluation and supports, the result 
will be a pipeline of principals able to improve teaching quality and student achievement district- 
wide, especially in schools with the greatest needs. 



In other words, the initiative aimed not only to achieve solid implementation in districts but also to test the 
power of the pipeline to affect student performance. The multi-year evaluation of the Principal Pipeline 
Initiative therefore has a dual purpose: to analyze the processes of implementing the initiative’s components 
in the participating districts from 2011 to 2015; and then to assess the results achieved in schools led by 
novice principals who had exposure to the initiative as implemented. A future evaluation report, scheduled 
for publication in 2018, will answer questions about the initiative’s effects on achievement. 

This report assesses the implementation of all components of the initiative as of spring 2015. Although the 
initiative continued to support the districts’ continuing work beyond 2015, this was a point at which all 
components were expected to be up and running. This report recapitulates and extends findings from previ- 
ous reports that described the districts’ plans and first-year activities (Turnbull, Riley, Arcaira, MacFarlane, 
& Anderson, 2013); analyzed the preparation and support for school leaders offered by districts and their 
partners (Turnbull, Riley, & MacFarlane, 2013); summed up the status of implementation as of 2014 
(Turnbull, Riley, & MacFarlane, 2015); and reported in depth on evaluation and support for novice princi- 
pals (Anderson & Turnbull, 2016). 


The primary purpose of the implementation reports is to inform local decision makers about the possibili- 
ties and challenges of following these districts’ example. Therefore this report identifies the districts’ pur- 
poses and strategies in each major part of their work, analyzes what supported or impeded their progress, 
and reports on early evidence of results from the perspective of district leaders and novice principals. It 
draws implications for districts and for the policymakers who support them. 


1 The initial grants ranged from $7.5 to $12.5 million. Supplemental grants in 2013 supported new efforts to strengthen the work of the district staff members who 
supervised principals. 


Background: Intent and Design of the Principal Pipeline Initiative 


After a decade of supporting research and program innovations in educational leadership, The Wallace 
Foundation saw research evidence that a principal has a pivotal role in a school’s instructional quality 
(Seashore Louis et al., 2010) and that urban districts have the power to reshape principal preparation and 
support (Orr, King, & LaPointe, 2010). 

Before and during the Principal Pipeline Initiative, research and analysis have continued to highlight the 
critical role of principals and possible ways of shoring up their capabilities. A statewide analysis of Texas 
data identified principal effects on student achievement (Branch, Hanushek, & Rivkin, 2012). Long-term 
work by the Chicago Consortium on School Research (Bryk, Sebring, Allensworth, Luppescu, & Easton, 
2010) identified school leadership as a critical support for school improvement, one that undergirds the 
other supports (parent-community ties, professional capacity, student-centered learning climate, and in- 
structional guidance). A report from Bain & Company (2013) identified needed innovations in pathways to 
school leadership and described ways in which some districts have carried out those innovations. The 
Council of the Great City Schools assessed the principal supervisor role nationally and in Principal Pipeline 
Initiative districts, with Wallace support, and recommended that the role be clarified and refocused on in- 
structional leadership (Corcoran et al., 2013). Principal evaluation has been the subject of research-based 
analysis and recommendations (Clifford & Ross, 2012; Davis et al., 2011). 


The initiative reflected the belief that districts could do much to develop and support principals as effective 
instructional leaders if they specified clear expectations for principals in the form of leader standards, re- 
shaped traditional preparation and hiring processes for aspiring principals, and restructured evaluation and 
learning opportunities for new principals. The major components of the initiative were expected to work 
together. Preparation, hiring, evaluation, and support would all be aligned to the same standards and thus 
would be mutually reinforcing. Key elements of each component would be the following; 



STANDARDS for leadership would specify the capabilities and performance that the district 
wanted in principals, and the district would shape all its policies and practices related to aspiring 
and novice school leaders around these standards. In addition to using the standards as criteria for 
hiring principals, districts would ensure that preparation programs would use the standards in ad- 
missions and in instruction; that the evaluation of new principals would assess the extent to which 
they met each standard; and that on-the-job support would specifically address any gaps between 
standards and performance. 


PRESERVICE PREPARATION programs for aspiring leaders, whether lodged in universities, non- 
profit organizations, or districts themselves, were required to meet several specifications based on 
the findings of a major Wallace-supported study of exemplary preservice programs (Darling- 
Hammond, LaPointe, Meyerson, Orr, & Cohen, 2007). Admissions criteria and program curricula 
would align with district standards. Drawing also on the findings of Orr and colleagues (2010), the 
initiative emphasized the importance of ongoing program improvement based on data about the 
quality of graduates’ preparation and, for programs based in universities or nonprofits, a partner- 
ship that would give the district a substantial role in shaping the program. 
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In HIRING AND PLACEMENT, the initiative called for “rigorous processes of selection and 
matching.” Selection procedures would be designed to collect and consider relevant information 
about capabilities aligned with district standards. Districts would assign new principals only to 
schools for which their particular strengths would be a good match. 

Finally, ON-THE-JOB EVALUATION AND SUPPORT for school leaders would also be aligned 
with standards. Districts would systematically evaluate novice principals’ instructional leadership 
capabilities, give them feedback, and, on the basis of the evaluation results, provide professional 
development and skilled mentoring to support them in further developing their needed capabilities 
(The initiative defined novices as those in their first, second, and third years in the position.) 
Districts would also provide professional development for assistant principals with potential for 
advancement. 


The initiative also brought the expectation that district policies and practices related to school leaders 
would build the district’s capacity to advance its educational priorities. All components would closely re- 
flect the district’s strategic priorities and would achieve district-wide scale and long-term sustainability. 

Based on the districts’ early experience, the foundation incorporated supporting mechanisms into the initia- 
tive design: 



The foundation supported development of a new data system, called a Leader Tracking System, as 
a repository for individual-level, longitudinal information about aspiring and novice leaders. The 
system would include data on the individual’s preparation and certification history, positions held 
and the schools in which he or she had held them, assessed skills and performance in each position, 
and participation in professional development and coaching. 

When the pivotal role of principal supervisors became evident through district experience and a re- 
port of the Council of the Great City Schools (Corcoran et al. 2013), the districts extended their 
work to include redesigning principal supervisor roles and training an expanded corps of principal 
supervisors, with the aim of more intensively supporting principals as instructional leaders. The 
foundation provided supplemental funds for this work. 

Similarly, the districts were encouraged to provide support for sitting assistant principals. As it be- 
came clear that most future principals would spend time in this role, the opportunities for learning 
and development in the position commanded greater attention. 


An important dimension of the initiative was active partnership between grantees and The Wallace 
Foundation. This included not only each district’s ongoing dialogue with senior foundation officials about 
detailed work plans and budgets, but also extensive opportunities to receive technical assistance from 
providers vetted by the foundation and to participate in the initiative’s Professional Learning Communities. 
The technical assistance included skilled help for each district in setting up its Leader Tracking System and 
in using Quality Measures, a suite of tools designed by Education Development Center with Wallace support, 


including indicators and rubrics for assessing leader preparation programs in a facilitated review of evi- 
dence. Professional Learning Communities were set up for district teams (including superintendents, local 
project directors, and others), for leaders of principal preparation programs, and for principals. The project 
directors also had opportunities to visit each other’s districts to learn about their peers’ work under the ini- 
tiative. 

In relying on large urban school districts to carry out the initiative, The Wallace Foundation took a risk. 
School districts, especially large ones, have a reputation for volatility in policy and a sad residue of shifting 
priorities over time (Hess, 1999; Hightower, Knapp, Marsh, & McLaughlin, 2002). They are also criticized 
for sluggishness in policy execution (Finn & Kanstoroom, 2000; Honig, 2006). The district is not a mono- 
lith that moves in an orderly fashion to carry out an external idea, but rather is a collection of subunits 
with histories, agendas, and incentives that collectively impede coherence (Spillane, 1998). In short, major 
cross-district initiatives have typically ended with pockets of partial implementation and a renewed convic- 
tion that big school districts are both fickle and cumbersome agents of change. It was possible that shifting 
local priorities would swamp the initial focus on principal leadership or that bureaucratic tangles would 
slow the work. 


In studying implementation of the Principal Pipeline Initiative, this evaluation assessed the extent to which 
the districts carried out policies and practices consistent with the initiative design. The four major pipeline 
components — standards, preparation, hiring, and evaluation and support — were central foci of the study. 
For each component in each district, we assessed what the districts had done to put new practices in place 
or strengthen what they were already doing. We analyzed what supported or impeded implementation, ac- 
cording to leaders in the district and preparation programs, and how new principals and aspiring principals 
perceived their experiences with each pipeline component. 



Evaluation Methods and Reporting 

We offer here an overview of data sources and data analysis for this study of implementation. More details 
appear in Appendix A. 


Data sources 

This report is based on an analysis of data collected by the evaluation team from the following sources: 

Semi-structured interviews with administrators in district central offices and in outside programs 
that partnered with the districts (e.g., in universities), conducted during annual site visits in spring 
2012-15. 

Focus groups with novice principals and assistant principals (i.e., those in their first, second, and 
third years in their position) in spring 2013-15. 
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Surveys of novice principals and assistant principals in spring 2013-15. 

Documents including the districts’ proposals, work plans, and progress reports for the foundation. 

Observation of and participation in cross-site meetings from 2011 through 2015, including obser- 
vation of presentations and panel discussions by district leaders. 

Site-visit interviews were arranged by the project director in each district, responding to specifications from 
the evaluation team. In each year we requested interviews with the project director, the superintendent (or, 
in New York City, another high-level official in the central office); other members of the executive team 
such as the directors of human capital, curriculum and instruction, and data systems; and central-office 
staff and partner-program leaders who were, collectively, knowledgeable about standards, preparation pro- 
grams, hiring and placement, supervision, evaluation, and support for principals. Where two or three peo- 
ple worked closely together on a particular function, the team typically conducted a joint interview with 
them. In some cases, project directors arranged one or more larger group interviews (e.g., the eight princi- 
pal supervisors in Hillsborough County were interviewed in two groups in 2014, as were the eight principal 
coaches in that district). Numbers of interviewees are shown in Exhibit 1. 


Exhibit 1: District and partner interviewees, by district, 2012-15 


District 

2012 

2013 

2014 

2015 

Charlotte-Mecklenburg 

14 

22 

11 

13 

Denver 

19 

21 

13 

22 

Gwinnett County 

14 

15 

16 

19 

Hillsborough County 

17 

18 

30 

18 

New York City 

8 

14 

18 

15 

Prince George's County 

19 

23 

16 

16 

Total 

91 

113 

104 

103 


Exhibit reads: In Charlotte-Mecklenburg, 14 district or partner staff members were interviewed in 2012. 



The semi-structured interview protocols included not only factual questions about all components but also 
probes for the respondents’ perceptions of what district practices appeared to be serving the intended pur- 
poses, what practices appeared to need improvement, and what changes, if any, were under consideration. 
Respondents were asked more detailed follow-up questions about their particular areas of responsibility. 




Focus groups with novice principals and assistant principals addressed their experiences and perceptions re- 
lated to standards, preparation, hiring, evaluation, and support. Project directors identified the participants 
for these focus groups. 

More complete data were gathered from novice principals and assistant principals in web-based surveys ad- 
ministered annually in spring 2013-15. For principals, all of those in their first three years in the position were 
surveyed in all districts. For assistant principals, all in their first three years in the position were surveyed in 
five of the districts; a random sample was surveyed in New York City. The surveys addressed these novice 
school leaders’ perceptions and experiences related to each pipeline component, along with a few questions 
about their backgrounds and career plans. Surveys were revised from year to year to provide more data on the 
focal topics for upcoming study reports; thus the 2013 surveys had a few extra questions on preparation ex- 
periences, and the 2015 surveys had a few extra questions on principal evaluation and support. 

Response rates generally exceeded 80 percent in five of the districts and ranged from 31 to 54 percent in 
New York City across the years. Appendix A provides information on the response rates and on our proce- 
dure for checking for non-response bias in the New York City principal data. (The New York response rate 
was lowest, 17 percent, among assistant principals in 2013, and we have excluded those respondents from 
our analysis.) 


Data analysis 

For this study of initiative implementation across six districts, the district is the primary unit of analysis. 
Therefore, where this report presents a single frequency drawn from responses across all districts, the 
analysis gives equal weight to each district. The raw survey responses, if compiled across districts, would 
over-represent New York City — which had the largest numbers of novice principals — and under-represent 
the other districts. Because of the significant cross-district differences in numbers of novice principals, we 
applied post-stratification weights as described in Appendix A so that each district would be equally repre- 
sented in overall analyses. 



The survey data presented in this report were analyzed for change over time in two main ways: (1) by 
comparing cross-sections of the principal and assistant principal responses gathered in 2013, 2014, and 
2015; and (2) by comparing cohorts of principals and assistant principals who started their position at 
five different time points. The first three cohorts of principals and assistant principals were surveyed in 
2013, and one new cohort of first-year principals and assistant principals was surveyed in each of the two 
subsequent years for a total of five mutually exclusive cohorts. Thus for several analyses in this report we 
compare the first two cohorts of principals who started their position in 2010-11 or 2011-12 to the last 
two cohorts of principals who started their position in 2013-14 or 2014-15. Our aim in focusing on these 
combined cohorts was to maximize the distance in time between the cohorts while boosting the numbers 
of cases by combining pairs of cohorts. See Appendix A for a more detailed description of cohort assign- 
ments and analyses. 
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Qualitative analysis was iterative. The evaluation team coded interview transcripts and notes according to 
the components and supporting mechanisms of the Principal Pipeline Initiative. Consistent with the purpose 
of the implementation study, the analysis for each component emphasized what had been done, what had 
helped or hindered those steps, what appeared to be working well, what challenges had emerged, and what 
district leaders wanted to do next. Multiple iterations of analysis identified and refined the specific themes, 
descriptions, and analyses presented in this report. Drafts were reviewed by the particular team members 
who visited the site for factual accuracy and revised as necessary. The districts’ progress reports to The 
Wallace Foundation were a supplementary source for detailed factual descriptions of policies enacted; simi- 
larly, the initial proposals were a source for facts about policies and practices in place before the initiative. 
Finally, project directors in the districts conducted a fact-check of this report’s text prior to publication. 


Organization of this report, and future reporting 

Chapter by chapter, this report analyzes the ways in which the six districts tried to improve principal stan- 
dards and competencies, preparation, hiring and placement, and evaluation and support. For each of these 
components it identifies key strategies across the districts, then reports on the accomplishments and remain- 
ing issues that were evident by spring 2015. An additional chapter analyzes a lingering puzzle that came 
into view over the course of the initiative: how to incorporate the assistant principal position into strategies 
for developing future principals. The report ends with a chapter drawing conclusions and implications from 
these findings on implementation. 

The overall Principal Pipeline Initiative evaluation will produce a final report on effects, analyzing achieve- 
ment and other data from schools led by principals who were exposed to pipeline components as imple- 
mented in the districts. That report is scheduled for late 2018. 



2. Principal Standards and Competencies 
Living Documents in Use 


r he first component of the Principal Pipeline 

Initiative, standards, became a driver of district 
action. Districts embraced the development and 
use of standards for principals and found these stan- 
dards helpful. Standards and competencies provided a 
powerful, aligned structure for principal evaluation, job 
descriptions, and some preparation programs. 

The Problem Facing Districts Initially: 

Standards Weren't Clear 
and Weren't Used 

Each district already had documents laying out stan- 
dards for school leadership at the start of the Principal 
Pipeline Initiative grants in August 2011, but the stan- 
dards lacked specificity and clout. In general, district 
leaders said they had a loose patchwork of language 
about school leadership that did not communicate what 
the district really wanted principals to know and do. 

The wording of local standards was often borrowed from a 2008 publication of the Interstate School 
Leaders Licensure Consortium and various state or local documents, and was not well known or much 
used in district practice. District leaders saw that they could use revamped standards in consequential deci- 
sions in principal hiring and evaluation. They wanted to develop clear, concrete, and actionable statements 
that could become a shared language about school leadership in the district, communicating behavioral ex- 
pectations rather than hazy exhortations. 


Takeaways 

1 . The use of standards was powerful in 
helping districts align their actions and 
policies to their priorities for school 
leadership. 

2. Principal standards shaped the design of 
preparation programs, hiring criteria, and 
on-the-job evaluation rubrics. 

3. Standards development or adoption was 
not a one-time event. The standards were 
living documents that districts continued to 
clarify based on experience. 

4. The wording of standards became a 
shared language about school leadership 
within districts, according to interviewees. 



Principal evaluation was an arena in which district leaders readily saw the potential value of clearly stated 
standards and competencies. They needed to define principal “professional practice” because their states 
required local educator evaluation to incorporate measures of professional practice. Moreover, district lead- 
ers believed that more specific feedback in the evaluation process could benefit principals, and they saw 
standards as a way to ensure that the feedback reflected district priorities. 
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Strategies: Putting an Early Version of Standards to Work, 
and Revising the Standards 

Enlisting broad-based committees in drafting standards 

Districts launched their work on standards quickly, consulting broadly with local stakeholders and often 
bringing in expert help. For example, Hillsborough County engaged several committees of principals and 
assistant principals in developing and refining initial lists of School Leader Standards and Competencies, 
then asked all principals and assistant principals to vet the resulting competencies. Experienced outside con- 
sultants helped with refinements such as streamlining the early draft. In Denver, a small committee of cen- 
tral-office staff and principals developed a first draft in 2011, reflecting state standards, competencies 
emphasized in a partner program of principal preparation, and national sources. District stakeholders then 
vetted the draft, and it was shortened with help from consultants. Charlotte-Mecklenburg was required to 
use state standards but formed a group of district and building leaders and preservice partners to identify 
key competencies for leadership practice in the district. They identified a subset of the state standards that 
the district prioritized, calling these their Super Standards. 


Revising the standards while using them 

The standards did not remain static. All the districts adapted and revised their standards and competencies 
whenever necessary to meet practical challenges, especially in hiring and evaluation. The continuous improve- 
ment was not designed as a structured, systematic process, but it took place in each district as issues arose. 

For example, after a 2011-12 pilot of a principal evaluation system based on the new standards, Denver and 
a consulting firm revised the standards by consolidating related standards and reducing the number of indi- 
cators from 22 to 12. Key purposes of those revisions were to clarify performance expectations and to place 
a greater focus on the leader standard for supporting English language learners, a district-wide priority. 

Charlotte-Mecklenburg first built an evaluation instrument and procedures, then rewrote the principal job 
description to reflect both the evaluation instrument and the standards. Then the district repeatedly fine- 
tuned recruitment and selection processes, with consulting help, in order to tighten their alignment with the 
job description. A district official said the revision of competencies and measurement rubrics benefited from 
the process of figuring out how candidates might demonstrate the skills and beliefs the district described: 

We really worked on [questions like,] for belief in students, what does that mean? To what level do 
we expect our people to demonstrate that, to be successful as a first-year principal? And so that is 
when we really began the redesign work and came up with our new competencies, new rubrics. 

Hillsborough County formed an Alignment Committee composed of the same individuals who had devel- 
oped the original standards and competencies, including principals and district-level staff. The committee 
and consultants used the competencies to drive the redesign of preparation programs, assessment, and sup- 
port at multiple career stages. 



When principal supervisors used a standards-based rubric 2 to evaluate principal performance, some identi- 
fied gaps and ambiguities in their leadership standards as well as in the rubric. A Denver district leader, for 
example, said that because principals were expected to develop assistant principals’ leadership capacity, this 
should be clearer in the rubric. District leaders said they heeded the feedback from principal supervisors 
and tried to convey a rubric’s meaning more clearly. Similarly, a leader in Hillsborough County said: “Now, 
... we’re going to do some word changes because as [supervisors] have used [the rubric], they’re saying, 
‘Can we change this word? Because it’s a little bit too ambiguous.’” 


New York City took a distinctive approach to the development and use of standards for principals. Its 
School Leader Competencies were reportedly not used well in hiring or evaluation as of 2012. A district 
leader said at that time: “We’ve been trying to figure out how to make better use of them, but also to 
strengthen them to speak more directly to what we expect our principals to do well.” Therefore the district 
launched an in-depth process of reframing the principal evaluation instrument to align closely with the 
Quality Review, which was a component of the district’s consequential evaluation system for schools. 


The Quality Review incorporated rubrics and indicators for assessing each school’s instructional core, sys- 
tems for improvement, and school culture. District leaders reasoned that new standards for principal evalu- 
ation should correspond to the qualities that the city expected to see in its schools. They set out, therefore, 
to map backward from the Quality Review Rubric to identify the principal competencies associated with 
each school-level indicator. Their plan was to use the resulting standards in principal evaluation and also in 
hiring and preparation. Building acceptance for this approach across offices, with New York State, and 
with the administrators’ union took time. By 2013, however, the new Principal Performance Review was in 
place, and new indicators aligned to the Quality Review Rubric had replaced the earlier School Leader 
Competencies in that process. 



Accomplishments , Challenges, and Lessons 

A mandate to develop standards might have led to pro forma compliance — composing a document and 
then shelving it. In these districts, however, the mandate led to active, continuing use of the standards. 
Consistent with the initiative design, districts aligned their principal job descriptions and evaluation rubrics 
to their standards and competencies. The curricula of some preparation programs, especially those run by 
the districts themselves, were also aligned to them. It seems that district leaders saw value in these uses of 
standards and therefore continued to pay attention to their standards documents. 


Standards drove coherence in policy and practice 

In interviews, district leaders reported that alignment with standards lent valuable coherence to their actions 
around the principal pipeline. For example, in 2015 a district official in Gwinnett County cited the active use 
of standards as an important result of participation in the initiative: 
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What’s different because of the PPI grant is that we have become more clear, more aligned, more 
cohesive with our leader standards. And evidence of that is that the leader standards today, are 
talked about, discussed, used in any number of learning situations. And before they were not. ... 
We weren’t as clear and as cohesive as I believe we are today. So today we ... talk about instruc- 
tional leadership with a much greater clarity than we did, say, three years ago. 

The director of a district-run preparation program in New York City commented in 2014 on the quality of 
learning opportunities for aspiring principals when their lessons were aligned with standards: 

This year, it was decided that we should align our content more closely to the Quality Review. . . . 
Our faculty also participates in the PPO [Principal Practice Observation] visits, which are the prin- 
cipal rating visits. Our faculty know exactly what people are being rated on.... Literally, one of my 
people can walk out of a PPO, ... go right to class, and say, “This is what you’re being required to 
do as a principal right now. This is what I’m seeing, and this is how you could do better.” 

In Hillsborough County in 2014, a person closely involved with the ongoing use of standards and competen- 
cies described their centrality in the district’s work with principals, assistant principals, and aspiring leaders: 

In year one I didn’t realize how important they were. We developed them, but now when we’re 
going into year three, the competencies drive everything we do. ...It’s the language that is now 
being used across the district. 

District officials were not the only ones in Hillsborough County to describe the competencies as “a lan- 
guage.” An aspiring principal said in a 2015 focus group: 

I think what the district views as a principal’s scope of responsibilities is very clear.... I’ve noticed 
there’s a real common language, too, used by all the district-level people when they talk to us, and 
what their expectations are, and what they hope for the future of the district, and where they want 
us to put most time, effort, and resources. And whether you’re talking to [a district staff member] 
or a [principal supervisor], or even our new superintendent, everybody’s talking the same language. 


Superintendent transitions might affect the stability of standards 

This initiative proved resistant to the effects of superintendent transition, although a change in top leader- 
ship could create temporary uncertainty around the use of standards. New York City had mayoral control of 
schools, and a new chancellor (the city’s term for superintendent) was appointed in 2015 by a newly elected 
mayor. In early 2015 our interviewees expressed uncertainty about the role of the Quality Review as an an- 
chor for principal evaluation going forward. However, in fact the standards for principals did not change. 



In Prince George’s County, abrupt leadership changes and impending changes in governance distracted de- 
cision makers from the transition to new principal standards, with the result that early implementation was 
bumpy. The end of an initial trial of a new standards-based evaluation system in 2012 coincided with the 
superintendent’s resignation. An interim superintendent led the district for the next year while the county 
executive sought and won legislation giving him authority over the district and reorganizing the school 
board. Under conditions of temporary instability, a small group of hard-working district staff refined the 
new standards, designed an evaluation system, and submitted the standards to the state for approval in 
2013. Principal supervisors and principals had little opportunity to learn about the standards in depth be- 
fore a new superintendent took office in June 2013 and the approved system had to be rolled out. In the 
following year, though, district leaders were able to work more intensively to solidify the supervisors’ un- 
derstanding of the standards. 

The cases of New York City and Prince George’s County illustrate the possibility of discontinuity associ- 
ated with either a superintendent who brings a new mandate or an unsettled interregnum between superin- 
tendents. In contrast, Charlotte-Mecklenburg and Hillsborough County had transitions that did not affect 
their work on standards. In each of those districts, the person named as interim superintendent and then as 
superintendent had already been closely involved with the process of standards development and use. 


Districts expressed satisfaction with their work on standards 

The standards were living documents that districts refined over the years. By 2012, just a year into the ini- 
tiative, interviewees commented that the initial draft standards had been too-long lists that needed stream- 
lining so that they could be of practical use. In later stages of the work with standards, gaps and 
ambiguities came to light when the districts used standards to shape principal evaluation or to anchor seri- 
ous conversations about the work of the principalship. Many interviewees who had been part of the revi- 
sion process said it had improved the clarity of the documents. Developing and using standards required 
attention and effort over time, but district staff reported that the work had been worthwhile. 


Summary 

After the initial editing of first drafts, all versions of the standards were usable tools. And after refinement 
on the basis of experience, the documents gave increasingly valuable structure and clarity to district policies 
on evaluation, hiring, and even principal preparation, according to district leaders. Some interviewees re- 
ported that standards and competencies for principals provided a shared language that was helpful to prin- 
cipals, aspiring principals, principal supervisors, and district leaders. It seems likely that using and revising 
the standards also gave district and school staff a chance to deepen their familiarity with them. 

The work that the districts did on standards fulfilled their initial hope of communicating specific expecta- 
tions for principal capabilities and principal performance. The standards helped ground the other Principal 
Pipeline Initiative components in practical terms, as we discuss in subsequent chapters. 
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3. Principal Preparation: 
Investment for the Long Term 


r he Principal Pipeline Initiative drew on research 
that identified desirable features for principal 
preparation (Darling-Hammond et ah, 2007; 

Orr et ah, 2010), but districts did not find preparation 
to be amenable to quick overhaul. They started the 
work quickly, redesigning or starting in-district prepara- 
tion programs and working with university and non- 
profit partners to make changes in existing programs, 
seeking greater alignment of preparation with their 
standards and more use of research-based features. 

However, seeing the results of this work among incom- 
ing principals would take time because several years 
typically elapsed from the start of preparation to the 
first day on the job as a principal. 

The Problem Facing Districts Initially: 

Preparation Was Largely Outside 
Their Control 

The six districts were eager to hire principals who 
brought strong skills and knowledge. Their direct power 
over principal preparation had limits as of 2011, how- 
ever. Individual educators did not need the district’s permission to pursue and obtain certification as school- 
building leaders by completing a state-approved program in a university or nonprofit organization and 
meeting other state-established criteria. Districts might have little or no authority over program designs. 

District leaders wanted a stronger role in program design and operations and in individual career choices. 
In order to manage preparation programs and opportunities, however, districts had to work with state re- 
quirements, the existing ecosystem of preparation providers, and limits on the resources available for new 
programs. All these limitations meant that any major structural change would be slow in coming. 

Any changes in principal preparation would take time to affect a district’s new crop of principals. Aspiring 
principals had a multi-year journey from the beginning of formal preparation to their first job as a princi- 
pal, a journey that often included a stint as an assistant principal after receiving initial certification. The 
median time that elapsed from starting preparation to becoming a principal was ten years in Hillsborough 
County, three years in New York City, and an average of six years across all six districts, as reported by 
first- and second-year principals in 2015. 


Takeaways 

1. The districts reshaped principal 
preparation through new or revamped in- 
house programs and data-informed 
partnerships with other preparation 
providers. 

2. District staff used data and their school 
visits to identify potential leaders. 

3. Districts wanted to offer authentic, on- 
the-job leadership experience during 
preparation, but they struggled to mobilize 
enough sites and mentors for such 
experience. 

4. Improving preparation was not a quick fix 
for the quality of new principals because 
principal preparation took years: among 
these six districts, the median time from 
starting preparation to becoming a principal 
ranged from 3 to 10 years. 
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Strategies: Starting Programs , Spotting Talent , and 
Bolstering On-the-Job Learning 


All the districts took action on principal preparation under the Principal Pipeline Initiative. In various ways 
they initiated programs and made data-informed program improvements, partnered with universities or 
nonprofits, stepped up their efforts to spot talent, and worked to strengthen clinical preparation. District- 
by-district specifics are summarized in Exhibit 2. 
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Exhibit 2: Summary of initial status and actions taken in 
principal preparation, 2011-12 through spring 2015, by district 


District 


Status as of 2011-12 


Key actions taken through spring 2015 


Charlotte- 

Mecklenburg 

Program partnerships in place with a university 
and a nonprofit. 

Nonprofit program including a year-long 
residency. 

Co-developed programs with two more 
universities. 

Phased down the nonprofit partnership. 

Piloted a district-run program. 

Denver 

Program partnerships with a university and a 
nonprofit. 

Participants being selected for a district-run 
program to start in 2012-13. 

University program including a year-long 
residency. 

Co-developed two more university programs. 

Operated the new district-run program, with a 
year-long residency (but allowing the participant 
to choose an internship in the home school 
instead). 

Gained authority from the state to issue 
principal licenses. 

Gwinnett 

County 

Mandatory district-run program for aspiring 
principals. 

Optional district-run program for aspiring 
assistant principals. 

Partnerships initiated with four universities. 

Added a semester-long residency to the district 
program for aspiring principals. 

Encouraged principals to hire graduates of the 
district program for assistant principals. 

Maintained four university partnerships and 
added a fifth. 

Hillsborough 

County 

Largely mandatory district-run program for 
aspiring principals. 

New, mandatory district-run program for 
aspiring assistant principals. 

Partnerships initiated with two universities. 

Increased selectivity of the district-run programs. 

Revised content of the district-run program 
curricula to align with district standards for 
principals. 

Maintained the university partnerships and added 
others. 

New York 

City 

Program partnerships with two nonprofits, with 
residencies. 

District-run program with a summer residency. 

Partnerships initiated with two universities. 

Accountability for the school support networks 
that included measures of their record of 
identifying prospective leaders. 

Reconfigured university partnerships: selected 
different universities to participate; shifted the 
focus of the partnership from changing 
coursework to changing the internship. 

Developed guidance for applicants and a common 
application process for partner programs. 

Prince 

George's 

County 

Program partnership with a nonprofit, with a 
residency. 

Partnership initiated with a university. 

Curriculum being developed, with a partner, for 
a district-run program to start in 2012-13. 

Added university partnerships for a more 
diversified preparation portfolio. Phased down 
the nonprofit partnership. 

Operated the district-run program, and increased 
its selectivity. 






Launching and strengthening programs 


All six districts started or expanded their own, district-run principal preparation programs for high-poten- 
tial assistant principals. Five districts did this in 2011-12, and Charlotte-Mecklenburg followed in 2014. 
Denver, Prince George’s County, and Charlotte-Mecklenburg initiated new programs, while Gwinnett 
County, Hillsborough County, and New York City worked to strengthen their existing programs. 

All of the districts worked with universities, heeding the advice of Wallace-supported research that had 
found that a program based in a university could have more staying power than one based in the district 
(Orr et al., 2010). Charlotte-Mecklenburg and Denver already had certification programs custom-designed 
and jointly led in partnerships with nearby universities. District officials were directly involved in recruiting 
and screening candidates for admission to the programs. These programs continued, and the two districts 
added other, similar partnerships. In each of the other four districts, top leaders initiated partnerships with 
nearby universities. All districts were selective in choosing partners, using criteria such as: philosophical 
alignment with the district’s beliefs about leadership competencies and how to develop them; willingness to 
communicate often; and openness to program changes. New York City and Prince George’s County estab- 
lished dedicated university programs for aspiring leaders, tailored to district standards and needs, with fel- 
lowship support for enrollees. Gwinnett and Hillsborough Counties formalized their relationships with 
selected university programs for administrator certification. District officials met regularly with the pro- 
gram leaders in the universities, seeking to influence program design and curriculum. 

Four districts also had prior agreements with selective nonprofit programs that offered an alternate route to 
principal certification, with a residency. New Leaders prepared principals for high-poverty schools in urban 
districts across the country including Charlotte-Mecklenburg, New York City, and Prince George’s County; 
the New York City Leadership Academy’s core mission at its founding in 2003 was to prepare principals 
for New York City, with an emphasis on principals for struggling schools; Get Smart Schools (now 
Catapult Leadership) offered a two-year program in Denver that placed its residents in charter schools, 
aiming to cultivate innovative leadership. The scope of most of these agreements with nonprofits shrank 
over the years of the initiative, however. Cost was a factor, as was the districts’ perception that they had 
improved their own capacity to design and carry out in-house preparation. 


Systematically assessing programs 

District and partner-program leaders pointed to ongoing, systematic discussion of program data as a con- 
tributor to the value of partnership arrangements. They used Quality Measures for Education Leadership 
Systems and Programs, a suite of tools developed by Education Development Center with Wallace support 
that includes indicators and rubrics for assessing leader preparation programs. It is organized around pro- 
gram components associated with program effectiveness (Darling-Hammond et al, 2007): course content 
and pedagogy; clinical practice; recruitment and selection; and three components related to graduate per- 
formance outcomes (knowledge, skills, and competencies; responsiveness to market demand; and impact on 
school, teacher, and student performance). The Principal Pipeline Initiative provided facilitators, training, 
and protocols for districts and program leaders to assess these program components, systematically gather- 
ing and rating the evidence. 
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The districts’ Leader Tracking Systems, developed over the years of the initiative, added to the store of data 
that could readily be used for data-based program improvement. As of 2015, use of the data in these sys- 
tems for ongoing program assessment was just beginning. 


Selectively encouraging individual aspiring leaders 

Another strategy involved spotting talent and guiding selected individuals — but not others — to growth op- 
portunities. Looking back in 2015, a top district leader said a goal of participating in the Principal Pipeline 
Initiative had been to change: 

... this whole [process of] individual people self-selecting themselves, going into leadership [programs], 
...when maybe they aren’t the right people to eventually be principals, and it’s allowing us to be very 
purposeful, very intentional, and taking [selected individuals] through a very supportive process. 

All districts encouraged principals to find and cultivate rising talent, and some expanded the circle of re- 
sponsibility for developing aspiring principals to include other district staff, including principal supervisors 
and leaders of district divisions such as curriculum and instruction or special education. District staff who 
visited schools were responsible for finding and encouraging potential future principals. A New York City 
official explained in 2012 how the networks and clusters (support teams that provided assistance to 
schools) were held accountable for identifying leadership talent. A leader in Charlotte-Mecklenburg spoke 
in 2013 of the responsibility of area administrators: “We see our zone executive directors as capacity 
builders and talent identifiers for future principals.” 

Districts that operated in-house preparation programs or collaborated closely with university programs 
worked to set a high bar, based on district standards, for admission to those programs. Gwinnett and 
Hillsborough Counties had two opportunities to screen would-be leaders in the admissions processes for 
their district-operated preparation programs, once with aspiring assistant principals and then with sitting 
assistant principals who aspired to be principals. A Hillsborough County leader explained the high stakes, 
saying that selection for its in-house programs offered “a pretty straight line to a hiring process” for a 
principalship. 

All six districts provided aspiring leaders with structured information about preparation opportunities to 
help them plan. In New York City, the district developed a page on its website with capsule descriptions of 
all the preparation programs in the district or with which it partnered, with links to their websites, and that 
page became the second-most widely viewed page on the district’s site. By 2015, the district had moved to a 
single application for multiple preparation programs operated by the district and its partner organizations 
and universities. 

Selective encouragement meant, inevitably, dampening the hopes of some aspiring leaders. In Prince George’s 
County in 2014-15, a district official offered an after-school session for aspiring leaders titled, “So You Want 
To Be an Administrator.” It was well attended. The speaker itemized all the needed credentials and each of 
many kinds of experience in schools, pausing after each one to ask audience members to raise their hands if 



they had that credential or that experience. When most hands were raised at each step, he urged the atten- 
dees to think critically about the way they were preparing to compete successfully for principalships. 


Strengthening on-the-job learning 

Research that informed the Principal Pipeline Initiative said that the clinical component of principal prepa- 
ration is critically important (Darling-Hammond et al., 2007). Three districts added ambitious clinical 
arrangements to their own programs: Gwinnett County moved the participants in its principal-preparation 
program into different schools for 90-day residencies; 3 Denver moved participants into different schools for 
a full-year residency; and Prince George’s County had developed a full-year residency program to start in 
summer 2015 for selected assistant principals who had completed its principal preparation program. In all 
the districts, too, one or more stages of preparation involved leadership-related projects that participants 
carried out in their own schools. 

In a shift in direction starting in 2014-15, New York City altered the terms of its agreements with partner 
universities. Rather than focusing on an effort to reshape the academic curriculum, the new agreements al- 
lowed the district to enhance internships. The district shifted funds out of fellowships and into support for 
mentor principals. A district official commented that this could meet needs of the universities as well as 
those of the district and participants: 

I think [the universities’] point of frustration was that they didn't have much power to ...hold prin- 
cipals accountable for authentically mentoring their aspiring leaders. ...We have more power to 
change [mentoring] because we can pay the principal to do it,... and we can require principals to 
come to some training. And we can also give the schools some money that really enables them to 
free the person up from their classroom teaching responsibilities so that they actually can engage in 
an internship. 


Accomplishments , Challenges , and Lessons 

District leaders believed that their role in preparation had benefits 

Looking at their own preparation programs and the closest partnerships with universities, district leaders 
saw reasons to think that their involvement with the programs was serving district purposes. In particular, 
they were glad that a selective admissions process gave the district a checkpoint for spotting high-potential 
leaders; district leaders who conducted class sessions had a chance to observe and size up the rising cohort 
of principal candidates; and program content tailored to the principalship routines and priorities of the dis- 
trict immersed aspiring leaders in those routines and priorities. 


3 "Residency" was the term used most often for placement in a new setting for an extended period (e.g., at least a month, and up to a full school year) as an 
apprentice leader. 
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Alignment with standards was a strength that district leaders cited in some preparation programs, espe- 
cially their own programs. Districts could use their leader standards both in admissions criteria and in pro- 
gram curricula. Hillsborough County’s work on alignment, for example, included use of the district’s key 
competencies in the criteria for admission to district-run preparation, and the alignment committee spent a 
year on program development, trying to ensure that at each stage of preparation the aspiring principals 
would have opportunities to develop those specific competencies. 


By 2015 only a minority of incoming principals had been through 
all-new preparation programs 

Our surveys can offer only partial evidence on the results of districts’ work on preparation because most of 
the survey respondents had begun their preparation before program changes took place. Among the first- 
and second-year principals surveyed in spring 2015, the median elapsed time from starting preparation to 
becoming a principal had been six years, averaged across districts. It was highest in Hillsborough County 
(ten years) and lowest in New York City (three years). This meant that when the two newest cohorts of 
principals looked back on the totality of their preparation experience in 2015, as our survey asked them to 
do, most were looking at experiences that had begun before the Principal Pipeline Initiative began in 2011. 
The initiative could have affected only part of their formal preparation. 

It is not surprising, then, that most responses to our survey did not show statistically significant change in 
reported experiences across these cohorts of principals. Responses were similar over time on questions 
about program design features (use of case studies, portfolios, problem-based learning, and field-based or 
action research projects). They were also similar with respect to the programs’ emphasis on instructional 
leadership and leadership for school improvement. 

However, one difference over time was statistically significant: new principals in the later cohorts gave 
higher ratings on the extent to which their preparation experience, overall, had given them “a strong orien- 
tation to the principalship as a career” (Exhibit 3). On that question, the mean among principals who 
started on the job March 2010 through February 2012 was 4.01 on a five-point scale, and it rose to 4.19 
for those who started on the job from March 2013 through February 2015 (t[619]=5.04, pc. 05). The per- 
cent of novice principals in those cohorts giving the highest response, “to a great extent,” is shown for de- 
scriptive purposes in Exhibit 3, along with a mean of the percentages across districts. 



Exhibit 3: Novice principals reporting preparation 
gave them a strong orientation to the principalship as a career 
to a great extent, by district and cohort 
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Exhibit reads: The percent of principals who reported that their leadership preparation training program(s) gave them a strong orientation to the principalship as a career “to a great 
extent” was 36 percent for principals who started on the job from March 2010 through February 2012 and 50 percent for principals who started on the job from March 2013 through 
February 201 5 in Charlotte-Mecklenburg. 


Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 
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Principals’ reports of their internship or residency experiences during preparation also did not show consis- 
tent changes across districts. However, some positive change was apparent in Denver (although the changes 
were not statistically significant): some key features of clinical experience received higher ratings from later 
cohorts of principals compared with the two cohorts who started on the job by February 2012 (Exhibit 4). 
This change in ratings occurred at the time when Denver instituted a district-run residency. 
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Exhibit 4: Denver novice principal reports on attributes 
of their internship/residency, by cohort 


I was closely 
supervised and 
assisted by 
knowledgeable 
school leaders 


I had responsibilities 
for leading, facilitating, 
and making decisions 
typical of an 
educational leader 


I was able to develop 
an educational leader's 
perspective on school 
improvement 



Percent 


■ To a great extent To a considerable extent 


Exhibit reads: Twenty-four percent of principals in Denver who had an internship/residency and who started on the job from March 2010 through February 2012 reported that they 
had been closely supervised and assisted by knowledgeable school leaders to a great extent, and 24 percent reported this was the case to a considerable extent. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 


Districts faced practical barriers to a robust clinical experience in principal preparation 

Improving clinical experience was a priority, but districts struggled to make it happen. Moving aspiring 
principals out of their schools for a residency in a different school could be disruptive and expensive, ac- 
cording to district leaders. The vacated position in the original school had to be filled in some way, poten- 
tially at a high salary. A cost-saving approach did emerge in Gwinnett County, though, as a district official 
explained: “We started with hiring part-time folks to backfill. That’s where the cost really came, and we’ve 
slowly gone to asking principals to repurpose people in their building to backfill.” For a teacher leader, 
temporarily filling the job of assistant principal could offer a chance to develop leadership skills while sav- 
ing the district the cost of a higher-paid part-time substitute for the assistant principal. 

An authentic leadership experience did not occur automatically but depended critically on the will and skill 
of the sitting principal in the intern’s or resident’s building, according to district and program leaders. 
According to district leaders and focus-group participants, some supervising principals readily included 








aspiring leaders in instructional leadership activities, but others gave them tasks such as bus and lunchroom 
duty or assessment logistics that provided limited opportunities for growth. Some internships or residency 
participants said they spent much of their time shadowing the principal or had little change in responsibility 
from their previous position. 

By 2013, several district leaders spoke of improving their communication with and training of mentor prin- 
cipals, hoping to influence the assignments they gave to aspiring leaders and to build the principals’ capac- 
ity as talent developers. One district leader prefaced a description of her plans by explaining, 

I haven’t done as good of a job at getting their principals in and telling them, “[Leader develop- 
ment] is your responsibility. This is what this candidate needs to learn. You need to make sure 
they have actual experiences and actually prescribe those things to them and do a better job de- 
veloping them.” 

In another district, a leader said of these mentor principals: 

We’ve started to invest a whole lot more time and attention to how we work with them. How do 
we support them? How do we help them understand what is the role of the mentor? How they 
should be creating opportunities for folks to learn on the job? 


Summary 

It is too early to determine the full effects of the districts’ investments in principal preparation. Creating or 
modifying programs took time. Finding effective and practical structures for internships or residencies was 
a work in progress. The districts did make progress at a structural level: they initiated and iteratively 
strengthened programs; they forged partnerships with universities whose institutional stability might im- 
prove the chances of stable programming; and they introduced and began to routinize ways of identifying 
and encouraging potential principals. In short, the districts launched strategies for intervention in principal 
preparation. They were continuing to refine program design and implementation as of 2015, however. 

Our survey data also lend an important perspective on the time horizon for improving the preservice prepa- 
ration of incoming principals. Interventions designed in 2011 could not shape the entire preservice experi- 
ence of new principals by 2015 — particularly when some of these interventions actually added early stages 
to a required sequence of preparation experiences. A more complete picture of the effects of changes in 
principal preparation will have to emerge over the coming years. 
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4. Principal Hiring and 
Placement: Quick Wins 


D istrict leaders were eager to improve the 

processes of principal hiring and placement. 

They moved quickly to develop and refine 
standards-based processes and gather more data about 
applicants. Over time, they built and began to use 
elaborate data systems in hiring, placement, and 
succession planning. 

The Problem Facing Districts Initially: 

Hiring Was a Pain Point 

Leaders in all the districts had at least minor concerns 
about their procedures for hiring and placement as of 
2011-12, and in four districts the concerns were press- 
ing. Top leaders said they felt some uncertainty that 
they were placing the best possible candidate in each 
school. Most reported an unsystematic process of filling 
vacancies one at a time and a lack of hard, usable data 
about candidates. Only in two districts, Charlotte-Mecklenburg and Gwinnett County, did leaders report 
having made good progress in their hiring and placement systems before the Principal Pipeline Initiative 
began. Charlotte-Mecklenburg had created selective “talent pools” for would-be principals and assistant 
principals, and expected to fill all open leadership positions from among the candidates who had cleared 
the bar for entry into these pools. Gwinnett required aspiring principals to participate in its selective prepa- 
ration program and take part in performance assessments before becoming eligible for principal jobs. 

Before grant award, the other four districts (Denver, Hillsborough County, New York City, and Prince 
George’s County) essentially posted open positions one at a time and considered applications from anyone 
holding the state-required certification. Hillsborough County preferred the candidates who had been in its 
in-house preparation program, and New York City had a “hiring pool” of license holders who had met a 
few additional requirements set by the state, but these criteria did little to alleviate concerns in either dis- 
trict. Two districts provide examples of the kinds of shortcomings that district leaders were eager to fix: 

In Denver in 2011-12, the hiring process was decentralized and ad hoc. A principal supervisor for a 
group of schools took the lead in inviting applications for each vacant or soon-to-be-vacant princi- 
pal position, identifying specific criteria for that position. Applications would then arrive from can- 
didates who saw the job posting. Districtwide, there was dissatisfaction with the effort and the 


Takeaways 

1 . Policies and procedures for principal hiring 
were not hard to change, and results were 
visible in survey responses. 

2. Planning for principal placement became 
a more systematic, data-informed process. 
New Leader Tracking Systems offered 
insights into rising talent and emerging 
leadership needs. 

3. New hiring procedures required 
candidates to demonstrate skills aligned 
with principal standards. 

4. Districts still found it hard to spot candidates 
with the needed interpersonal skills. 

5. Success in placing principals was 
increasing, according to principals' reports: 
novice principals more often reported that 
their skills were an "excellent" fit with their 
schools in 2015 than in 2013. 
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results. Time was spent in screening unqualified applicants; a single candidate might go through 
half a dozen school-by-school selection processes; and it might turn out that the candidate placed 
as principal of one school would have been a better fit for a different school that happened to post 
its vacancy later. 


Prince George’s County relied heavily on interviews to assess candidates, a process that took time 
and was widely described as “it’s who you know.” District leaders were also concerned that the 
lack of specific feedback to applicants meant that aspiring principals would simply continue to 
apply for vacant positions without having a good opportunity to develop the skills they needed. 


Strategies: Using Data in Succession Planning and 
Multi-Stage Candidate Selection 

Districts eagerly implemented a set of procedural changes in hiring. They introduced longer-range planning 
around possible vacancies, preliminary selection stages for candidates, and simulations designed to reveal 
candidates’ strengths and weaknesses. By 2015 they were in the early stages of using their Leader Tracking 
Systems to support data-informed hiring and placement. Each district’s major changes in hiring are shown 
in Exhibit 5. 


Lengthening the time horizon for succession planning 

Changes made in Denver illustrate a systematic approach to succession planning, in contrast to the previ- 
ous ad hoc system of reacting to vacancies when they arose. Under the new system, top leaders and princi- 
pal supervisors began identifying specific vacancies and projecting an overall number of vacancies in the 
fall — something they had not done before. They also discussed assistant principals and others who might be 
candidates for leadership roles or further preparation. Development of a hiring pool complemented this 
work of identifying vacancies and possible candidates well in advance. A rolling process of application re- 
view produced a principal pool of dozens of candidates by spring. As vacancies arose, the district quickly 
filled some from the pool, and district leaders were generally confident in late spring that they had a good 
range of candidates ready for the likely vacancies. 



Exhibit 5: Summary of initial status and actions taken in principal 
hiring and placement, 2011-12 through spring 2015, by district 



District 

Status as of 2011-12 

Key actions taken through spring 2015 

Charlotte- 

Mecklenburg 

Selective hiring pool from which principals were 
to be hired. 

Centralized selection for the hiring pool. 

Finalists for specific school vacancies 
recommended by principal supervisors. 

Added practical demonstrations in the selection 
process for hiring pool and principal positions. 
Principal supervisors were trained in rating 
applicants. 

Began using Leader Tracking System in hiring and 
placement. 

Denver 

Candidates identified for each vacancy by 
principal supervisors, who developed job 
descriptions and reviewed applications. 

Instituted a hiring pool. 

Succession planning engaged principal supervisors 
and top leaders in discussing anticipated vacancies 
and candidates. 

Added practical demonstrations in the selection 
process for hiring pool and principal positions. 
Principal supervisors were trained in rating 
applicants. 

Began using Leader Tracking System in hiring and 
placement. 

Gwinnett 

County 

Practical demonstrations used in the selection 
process for principal positions and for the 
prerequisite district-run preparation program. 

Began using Leader Tracking System in hiring and 
placement. 

Hillsborough 

County 

Hiring based on interviews with top district 
leaders, with newly standardized procedures. 

Succession planning engaged principal supervisors 
and top leaders in discussing anticipated vacancies 
and candidates. 

Added practical demonstrations in the selection 
process for principal positions and for the 
prerequisite district-run preparation program. 

Began using Leader Tracking System in hiring and 
placement. 

New York 

City 

A hiring pool including all candidates who held 
the appropriate license and had completed 
simple performance tasks. 

Identification of candidates by school support 
networks. Selection by regional superintendents. 

Required practical demonstrations on skills 
aligned with standards for those in the hiring 
pool. 

Made the Leader Tracking System available to 
principal supervisors for use in hiring and 
placement. 

Prince 

George's 

County 

A new system of practical demonstrations 
aligned with new principal standards, replacing a 
previous system that relied on recommendations 
and interviews. 

Refined the components of the selection process. 

Began using Leader Tracking System in hiring and 
placement. 
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Introducing more ways to assess candidates' skills 

Prince George’s County was a district that moved quickly to revamp its hiring procedures, seeking a new, 
data-informed basis for hiring. Recommendations and “popularity” were outweighed by new measures 
that offered promise of greater objectivity and precision, and that were aligned with standards. A district 
official explained in 2012: 

People must demonstrate that they have the ability to carry out standards — they conduct an obser- 
vation, they analyze data, they create a vision statement. All these things are a part of the work 
that has emerged from the standards. 

Other districts, too, added steps to their assessment procedures for aspiring leaders, requiring them to 
demonstrate skills aligned with local leader standards. Consulting firms with expertise in human capital 
systems in education helped in developing customized tools and procedures. By 2015, assessment proce- 
dures in all districts included practical demonstrations, typically with simulated scenarios. Several assess- 
ments would occur over a day or more. They could include: 
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Role play: the candidate takes the role of principal responding to a difficult semi-scripted scenario. 

In-basket exercise: the candidate prioritizes and addresses a set of tasks and messages that might 
arrive in an in-basket. 

School data review: the candidate receives a package of school data and makes recommendations 
for data-informed improvement in that school. 

Teacher observation and feedback: the candidate observes a video of teacher instruction and pro- 
vides the written or oral feedback that he or she would give to the teacher. 


Building a Leader Tracking System to provide insight into rising talent 

Districts welcomed the chance to use more data in hiring, a process facilitated by the development of a 
Leader Tracking System that captured individual, longitudinal data on experience, performance, and as- 
sessed competencies. With Wallace Foundation encouragement and financial support, districts pulled to- 
gether the pieces of data that had resided in separate data systems or on spreadsheets. This was laborious 
work in most cases, but consultants helped with system design and development. Examples of data that dis- 
tricts began using in hiring and placement included: relevant student achievement data as a teacher and as- 
sistant principal; on-the-job evaluation scores as a teacher and assistant principal; experience with certain 
types of schools and students; language skills; performance on selection and exit assessments in preparation 
programs; and measured competencies related to instruction and leadership. 



As data on candidates increasingly arrived in a standard, user-friendly format, decision makers could find 
and use relevant facts more readily. In Prince George’s County, the summary information was called “the 
baseball card.” In Hillsborough County, a district leader said in 2015 that decision makers were “using the 
tool.... Not only do you get information about the candidate, you can do a comparison report. So say 
you're down to five candidates that really all seem good, you can actually click on a comparison report and 
see them all side by side.” Thus, for example, district leaders said in interviews that they might look for a 
candidate who had experience in particular grade levels or with English language learners and whose meas- 
ured competencies matched the needs that they saw in the school. None claimed that matching was becom- 
ing an exact science, but they welcomed an easier process of spotting candidates whose backgrounds they 
considered promising matches for a school’s most visible needs. 


Adding selection stages 

By 2014, all districts had selection stages and assessments that preceded — sometimes by several years — the 
hiring of a principal to fill a vacancy. One key purpose was to ensure a pool of high-quality candidates by 
using multiple stages of scrutiny. In Gwinnett and Hillsborough Counties, would-be principals faced three 
selection gates before they could apply for principalships: they had to be selected for the district’s assistant 
principal preparation programs, then selected as assistant principals, and then selected for the district’s 
principal preparation program. Denver and Prince George’s County instituted selective pools for aspiring 
principals, which was a step that Charlotte-Mecklenburg had taken in 2011. New York State tightened the 
requirements for entry into the hiring pool statewide, and New York City introduced new assessments for 
candidates who were in the pool, not to disqualify anyone but to inform hiring managers’ decisions with 
additional data. 

Systems of selection into a prerequisite program or talent pool also promised to streamline the process of 
filling principal vacancies by limiting the number of potential applicants for each vacancy. This was a prior- 
ity. Over the years, staff in some districts complained about a deluge of applications for each open position 
if they said they had not done enough to limit eligibility — or expressed relief that they had stemmed the tide 
through limits on eligibility. District managers in Prince George’s County cited their vivid memories of the 
numbers: “We put out a principal posting and over eight hundred applied,” said one. 


Accomplishments , Challenges, and Lessons 

Changes in hiring and placement offered quick wins for districts 

Unlike a change in the preparation component of the pipeline, a change in hiring and placement procedures 
could immediately affect the next cohort of incoming principals. This was evident in new principals’ survey 
responses about simulations as part of the hiring process (Exhibit 6). Among the two earliest cohorts of 
principals surveyed, who started on the job in 2010-11 and 2011-12, these practical demonstrations had 
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been part of the application experience (in applying for a talent pool or for a principal position) for 31 
percent, across districts. The rate of practical demonstrations rose to 59 percent for the two latest cohorts, 
those who started on the job in 2013-14 or 2014-15. This was the selection procedure that showed the 
greatest increase over time. 4 In terms of a weighted overall response across districts, the change was statis- 
tically significant. 
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Exhibit 6: Novice principals reporting participation 
in practical demonstrations in the principal application 
and hiring process, by district and cohort 
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Exhibit reads: Participation in practical demonstrations as part of their district’s formal application and hiring process was reported by 38 percent of principals who started on the job 
from March 2010 through February 2012 and 77 percent of principals who started on the job from March 2013 through February 2015 in Charlotte-Mecklenburg. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 


District leaders communicated a noticeable sense of excitement when interviewed about the hiring process. 
Superintendents whose districts had made changes in information systems or succession planning said they 
were enthusiastic about those changes. Other interviewees also reported that performance assessments ap- 
peared to provide a more authentic understanding of candidates’ competencies, compared with the previ- 
ous interview-based hiring procedures. A Denver principal supervisor interviewed in 2014 summed up the 
benefits of instituting a talent pool: 


4 Our survey data do not specifically identify the time(s) when a particular respondent applied to a hiring pool or applied for a principalship. We use the time of 
becoming a principal (i.e., the earliest two cohorts vs. the latest two cohorts) as a rough proxy for the time when groups of respondents experienced the hiring process. 

















Looking at the principal pool, we have had several changes and have finally gotten to giving us bet- 
ter options for principal candidates. The screening process for candidates to enter into a pool has 
greatly increased the level of expertise and quality of principals. Doing all the pre-work up front 
has led to higher-level candidates. 

The process of feedback to applicants also improved, addressing another problem that had initially con- 
cerned district leaders. About two-thirds of the novice principals who started on the job from early 2013 to 
early 2015 reported that the hiring process gave them useful feedback. This represented an increase over 
time, especially at the stage of selection into a hiring pool. 5 Among all novice principals who had applied to 
a hiring pool, the later cohorts were somewhat more likely than an earlier cohort to say that their profes- 
sional practice had been informed by feedback from that application, although the change was not statisti- 
cally significant (Exhibit 7). Perceptions of feedback at the hiring stage showed less change but remained 
relatively positive, rising from an average of 63 percent of respondents to 69 percent, across districts, agree- 
ing that it had helped inform their practice (Exhibit 8). 


Exhibit 7: Novice principals agreeing that as 
candidates for a hiring pool they received useful feedback 
for their professional practice, by district and cohort 
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Exhibit reads: Among Charlotte-Mecklenburg principals who had applied for a hiring pool, 23 percent of those who started on the job from March 2011 through February 2012 
strongly agreed or agreed that they received feedback from that application process that was useful for purposes of informing their professional practice; 43 percent of those who 
started on the job from March 201 3 through February 201 5 strongly agreed or agreed. 

Source: Principal Survey for “Evaluation ofthe Principal Pipeline Initiative,” 2013, 2014, and 2015. 


5 Because this question was asked for the first time in 2014, we do not have responses from the principals who started on the job in 2010-1 1; that cohort was surveyed 
only in 2013. 
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Exhibit 8: Novice principals agreeing that as 
candidates for a principal position they received useful feedback 
for their professional practice, by district and cohort 
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Exhibit reads: The percent of principals who strongly agreed or agreed that they received feedback that was useful for purposes of informing their professional practice as a candidate 
for principal positions was 58 percent for principals who started on the job from March 2011 through February 2012 and 59 percent for principals who started on the job from March 
201 3 through February 201 5 in Charlotte-Mecklenburg. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 


Novice principals' perceptions of their fit with their school improved 

Districts seemed to get better at placing principals where they were needed. Principals who started on the 
job at a time from March 2013 through February 2015 were more likely than earlier cohorts to self-report 
an “excellent” fit with the school in which they were placed (Exhibit 9). The increase in the weighted re- 
sponse across districts was statistically significant. It is possible that this change reflected improvement in 
the districts’ processes for screening and placing principals, which changed over the period of our surveys. 
Research suggests that this perception of fit could be a leading indicator of retention and success on the job 
(Kristof-Brown, Zimmerman, & Johnson, 2005). 


















Exhibit 9: Novice principals reporting an “excellent” fit 
of their skills, experiences, and interests with 
the needs of their school, by district and cohort 
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Exhibit reads: The percent of principals who characterized the fit between their skills, experiences, and interests and the needs of the school where they are principal as “excellent” 
was 62 percent for principals who started on the job from March 2010 through February 2012 and 77 percent for principals who started on the job from March 2013 through 
February 201 5 in Charlotte-Mecklenburg. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 


The changes in hiring and placement required investments of staff time 

The changes made in selection procedures and data systems took nontrivial amounts of staff time, both ini- 
tially and on a continuing basis, according to interviewees at the district level. In Charlotte-Mecklenburg, 
for example, several senior district staff members attended each talent-pool candidate’s multi-hour session 
of performance assessments and interviews. As one participant put it, “It has been very time-consuming.... 
I’m not going to sugar-coat that. We’re going to see if all that time commitment is worth it.” In New York 
City, a small and dedicated group of central-office staff and willing principals rated the performance of 
principal applicants, and the strain associated with the time commitment was clear when we interviewed 
group members. 

For Leader Tracking Systems, much work had to take place in order for individual data to be accessible. 
Designing and populating a Leader Tracking System took time and required the expertise of specialists 
whom the districts employed or retained as consultants, as well as assistance and expert review from 
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consultants whom The Wallace Foundation engaged. In each district there were problems to be solved in 
pulling the data together and designing user interfaces. In New York City, for example, retrieving data from 
multiple data systems was a major challenge in the years from 2012 through 2014, according to intervie- 
wees working on that task. 

User-friendliness was another challenge. District information-technology staff spoke of “being able to talk 
across the user and technical perspectives” in the effort to find and organize the most relevant data. Focus 
groups with prospective users began to clarify ways of organizing the information. Still, a staff member 
said of those prospective users, “When they say what they want, there may be a hundred decisions [still 
needed] about what they really want.” 


District leaders reported positive perceptions of their new principals, but shortages of 
some needed skills and dispositions 

District leaders were asked in 2014 and 2015 to provide an informal assessment of their latest cohorts of 
novice principals. In general, they spoke very positively. They especially praised the new principals’ knowl- 
edge of instruction. 

Some kinds of schools continued to pose challenges for leadership selection. It was common for high 
schools to draw too few strong candidates. Leaders in all districts also acknowledged challenges in finding 
the right candidates for high-need schools. These could include schools with low student achievement or 
low family incomes, or schools in which parents of high socioeconomic status made many demands on the 
principal. District leaders in Charlotte-Mecklenburg and Hillsborough County said in 2014 that they had 
seen both high-achieving and low-achieving schools pose leadership challenges, and that they still did not 
have as many strong candidates as desired when a vacancy arose in a school of either kind. 

In all districts, top officials expressed concerns that their hiring systems had not done enough to spot those 
leaders who could effectively build relationships, a positive culture, and a shared sense of commitment 
among the adults in their school. The following comments about incoming principals, each from a different 
district, were typical: 

They’re very passionate people. They get into the school ready and passionate to make a change, 
and it’s like they’re coming in with a sledgehammer and making change, but . . . they don’t realize — 
it’s personal leadership skills, they’re missing the boat. No one’s following them if they’re going in 
with a sledgehammer. So how do they navigate through that politically?... Are there ways they can 
navigate it better so it’s not as disruptive a change? 

I think sometimes you have to remind them about relationship building. They go in, and sometimes 
they go head on, tackle things. They have to remember they’re not part of the family yet. 



How do you use your positional authority? How do you read the culture of a school? ... I would 
say more of our principals struggle . . . over their human relations and inability to get the work 
done through and with people. 

With this concern in mind, district leaders said they wished they had better ways to measure emotional in- 
telligence (the term used in Denver) or “micro-political” skills (the term used in Charlotte-Mecklenburg). 
They regretted having placed some principals whose relationship-building skills turned out to be a poor fit 
for their schools. 


Summary 

Top district leaders viewed the changes in hiring and placement processes as successes. A change such as the 
greater use of simulations for screening candidates could take effect quickly. With systematic consideration 
of likely vacancies and a pool of vetted aspiring leaders, a district could make more thoughtful matches. 
Data in the Leader Tracking Systems became available for use in hiring and placement. Even the remaining 
challenges of finding strong candidates for high schools and for high-need schools were at least coming into 
clearer focus thanks to more data becoming available earlier. 

New principals’ perception of their fit with their school improved during the study period, possibly signal- 
ing that success on the job was more likely. In general, too, district leaders spoke highly of the capabilities 
they saw in incoming principals by 2014-15. They wished that they had better ways of anticipating the 
challenges that a particular candidate would face in forging a strong bond with the teachers and commu- 
nity in a particular school. However, they reported increasing confidence that their new principals’ skills in 
instructional leadership were strong. 
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5. Evaluation and Support: 
for Novices 

D istricts were interested in using systems of eval- 
uation and support as vehicles for improving 
new principals’ performance. All the districts 
quickly developed evaluation measures and procedures; 
most revised these over time; and all worked to build 
the capacity of principal supervisors as key contributors 
to both evaluation and support. Principal supervisors, 
mentors, and coaches generally provided valuable sup- 
port, according to principals’ survey responses. 

Professional development, however, remained an area of 
some disappointment. 

The Problem Facing Districts Initially: 

Some Novices Were Struggling 

At the start of the initiative, district leaders said that evaluative ratings of new principals’ strengths and re- 
maining weaknesses should be linked to more robust systems of professional learning and tailored support. 
No district wanted to use evaluation as a means to weed out larger numbers of novice principals, and in 
fact leaders in three districts (Denver, New York, and Prince George’s County) were concerned about high 
rates of voluntary or involuntary principal departures. District leaders wanted to see more novice principals 
succeed and remain on the job, and they believed that improved systems of evaluation and support could 
help. They also believed that evaluation rubrics should be aligned with their standards of principal profes- 
sional practice. The following statements from district leaders in Denver and Prince George’s County, re- 
spectively, in 2012 typified this view: 

The old evaluation system didn’t clearly define what the instructional leadership components 
looked like. We’re taking big steps forward, putting a stake in the ground around those compo- 
nents, with more clarity about overall performance. The old structure didn’t have final rating cate- 
gories. You’d just be renewed or non-renewed. 

We have had the same principal evaluation for years and years. It was perfunctory and didn’t get at 
the leadership standards we wanted. 

With respect to support, district leaders recognized that the job of principal could be a lonely one and that 
more skilled, experienced guidance and coaching might help. About two years into the initiative, a multi- 
district study by the Council of the Great City Schools, commissioned by The Wallace Foundation, con- 
cluded that principal supervisors in large urban districts often lacked the time and training to provide 
substantial support to principals (Corcoran et al., 2013). 



Takeaways 

1 . The roles of principal supervisors changed. 
Supervisors took on pivotal roles in on-the- 
job evaluation and support for principals, 
and districts worked to build supervisors' 
capacity. 

2. Most new principals reported in surveys 
that evaluation systems were accurate and 
informed their practice. 

3. A large majority of new principals valued 
the support from supervisors, mentors, and 
coaches, according to survey reports. 

4. Tailoring professional development to 
principals' individual needs was a continuing 
challenge for districts. 
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Exhibit 10: Summary of initial status and actions taken in novice principal 
evaluation and support, 2011-12 through spring 2015, by district 


District 

Status as of 2011-12 

Key actions taken through spring 2015 

Charlotte- 

Mecklenburg 

Specific, state-required standards used in 
evaluation system. 

A five-year induction sequence with district 
coaches and partner programs. 

Continued to use state standards in evaluation. 

Hired more principal supervisors and trained them 
in using the evaluation system and in coaching. 

Continued and refined the five-year induction 
program. 

Denver 

Principal retention based on school performance 
and conditions. 

Pilot use of an evaluation system based on 
emerging leader standards. 

Principal mentors for novices, consistent with 
state regulations. 

Developed standards and a new evaluation system 
specifying measures of student growth and 
professional practice, as required by the state. 

Hired more principal supervisors and trained them 
in using the evaluation system. 

Added executive coaches. 

Gwinnett 

County 

A longstanding evaluation system measuring 
student growth and principal practice. 

Expanded corps of retired principals as part-time 
mentors for novices. 

Fine-tuned existing evaluation measures. 

Hired more principal supervisors and trained them 
in using the evaluation system. 

Refined mentor roles, materials, and processes. 

Hillsborough 

County 

A new state-approved evaluation system. 

More principal supervisors recently hired, using 
the new evaluation system and expected to 
provide coaching in instructional leadership. 

A new district position of principal coach. 

A new, two-year principal induction program. 

Developed new principal standards and changed 
the evaluation system to align with them. 

Trained the principal supervisors in using the 
evaluation system and in providing feedback. 

Added and trained more principal coaches to 
work with supervisors to provide support. 

Developed cross-division central office team for 
support. 

New York 

City 

Principal retention based on school performance 
and conditions. 

Pilot use of a new principal evaluation process. 

Individual coaching for first-year principals from 
a partner nonprofit, with optional second year. 

Support from voluntary networks. 

Developed and rolled out a principal evaluation 
rubric aligned with the Quality Review used in 
rating schools. 

Trained principal supervisors in using the 
evaluation system. 

Piloted internal coaching. 

Prince 

George's 

County 

Principal retention based on school performance 
and conditions. 

Limited feedback to principals. 

Mentoring from sitting principals who received 
formal mentor certification. 

Developed standards-based evaluation system 
specifying measures of student growth and 
professional practice, as required by the state. 

Hired more principal supervisors and trained them 
in using the evaluation system. 

Added two full-time principal coach positions, and 
began coordinating their work with that of 
mentors. 





Strategies: Creating Standards-Based Evaluation and Systems of Support 


Over the years, the districts modified their systems of evaluation and support for principals. They intro- 
duced measures of principal professional practice (a step that was required under federal waivers of 
Elementary and Secondary Education Act accountability provisions, which all six states had received). In 
implementing their evaluation systems, they tried to ensure accurate, consistent measurement. They also 
worked to improve supervision and support, linked to evaluation results (Exhibit 10). 


Measuring professional practice with instruments based on standards 

Every district designed or adapted an evaluation rubric for supervisors to use in rating principals on the 
district’s leadership standards and competencies. Evaluation of principal professional practice was a state 
requirement in all six states. The areas that the rubrics commonly addressed were instructional leadership, 
human resource management, school climate and culture, planning, and school management and operations. 

Five districts made gradual changes to their evaluation tools and procedures before and during implementa- 
tion of the Principal Pipeline Initiative. Charlotte-Mecklenburg was a partial exception because the state of- 
fered little flexibility and the district moved quickly to apply the state standards, but discussion and 
training around the use of these standards was ongoing in that district during the initiative. The typical 
practice among these districts was an initial year of pilot implementation of a system, followed by (1) con- 
tinuing fine-tuning of the principal standards in consultation with the state; and (2) years of working with 
the principal supervisors who evaluated principals, aiming to familiarize them with the system, to gather 
their advice on improving the rubrics and procedures (as discussed in chapter 2), and to develop their skills 
in giving feedback. 

Each district designed the evaluation system to unfold something like a conversation between principals 
and their supervisors. The systems had procedures for goal setting and evidence gathering crafted to ensure 
ample opportunities for two-way discussion. Feedback for principals accompanied the process of data col- 
lection in hopes of supporting principals’ professional growth. Final performance ratings then directed 
principals’ attention more formally to areas of needed improvement. 


Redefining the job of principal supervisor 

The districts added more supervisors so that each could work with fewer principals, and they worked to 
change the jobs of principal supervisors. District leaders wanted supervisors to spend less time overseeing 
compliance and operations and more time helping principals succeed as instructional leaders. 

Reducing supervisors’ caseload of principals so that they could effectively support practice improvement 
was a district priority everywhere except in New York City, where the large number of principals made a 
major reduction in supervisors’ caseload impractical. Denver, Gwinnett County, and Prince George’s 
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County had added supervisors before 2014 in order to reduce each supervisor’s caseload. The Wallace 
Foundation gave each Principal Pipeline Initiative district additional funding ranging from $430,000 to $1 
million in 2014, as seed money to redefine and strengthen the position. Districts provided training to help 
supervisors deliver effective feedback, structure their school visits, and coach principals. 

Supervisors in all districts spent considerable time in schools. They were expected to collect a variety of evi- 
dence from a variety of sources to document and rate principals’ professional practice. Generally, this 
meant school visits that involved discussions with principals, teachers, and other school staff; observations 
of principal performance, including principal interactions with school staff; and reviews of artifacts, includ- 
ing plans for school improvement and teacher professional development. 

All districts worked with their principal supervisors on calibration, that is, an effort to increase consistency 
and fairness in the principal evaluation system. New York City took a systematic approach with evaluation 
simulations. Supervisors reviewed videos and artifacts and assembled evidence to rate each indicator of 
leadership practice. They shared with each other their evidence and rationale for a particular rating, and 
members of the group discussed whether they agreed. As a district official described it, the purpose was to 
raise and address questions about consistency: “Are we on the same page? Are we developing a consistent 
lens for looking at evidence through the rubric? Are we providing a coherent and consistent experience for 
the field when we go out there?” 
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Surrounding principals with support 


In each district, mentors or coaches offered support to novice principals. 6 In 2014-15 across the six dis- 
tricts, 90 percent of first-year principals, 74 percent of second-year principals, and 38 percent of third-year 
principals had a mentor or coach. The figures had not changed significantly since 2012-13. The overall per- 
cent of novice principals in these districts who reported that they had mentors or coaches in 2014-15 was 
69 percent, which was somewhat higher than the 62 percent among all principals in urban districts in 
2011-12 on the federal Schools and Staffing Survey (Manna, 2015). Districts varied in the type of support 
offered, along a continuum from helping principals learn to think for themselves to assisting them in the 
day-to-day management of their schools. 


Several districts enlisted nonprofit or university partners in developing the skills of their mentors or coaches 
through programs of formal training. District leaders explained that they recognized that helping a novice 
principal requires specialized skills, and that training can systematically build those skills. 

Many mentors and coaches said in interviews that they saw themselves as trouble-shooters, helping princi- 
pals address the day-to-day problems that arose. Gwinnett County worked to systematize the work of its 
mentor corps, setting up logs organized around the principal standards on which mentors recorded what 
skills or tasks they had worked on in each visit to a principal. 


6 "Mentoring" and "coaching" did not have consistent definitions across districts. In this report we use "mentors or coaches" and "mentoring or coaching" when 
presenting cross-district findings but use the local term when analyzing the supports in a particular district. 



Hillsborough County and Denver deployed support for principals in networks of six to ten specialized dis- 
trict partners. Hillsborough, for example, developed Area Leadership Teams in 2014 to help principal su- 
pervisors better support their principals’ needs. The team could include the principal coach, the elementary 
generalist, curriculum experts in reading, writing, math, and science, the special-education supervisor, the 
professional development expert for an area, a Response to Intervention 7 facilitator, and a human-resources 
representative. 

Several district leaders described developing principal professional learning communities that meet regu- 
larly to discuss problems of practice and receive support from district staff — such as instructional facilita- 
tors or assessment specialists — to develop or strengthen a skill or competency. District leaders led formal 
sessions and often set the agendas. 

Charlotte-Mecklenburg had a structured three-year induction program for novice principals at the start of 
the initiative, which it quickly expanded to a five-year sequence. District coaches worked with principals in 
groups; nearby universities and nonprofits provided specialized programs year by year. 

Principals in all districts also had an array of professional development options available. Some districts of- 
fered professional development courses and workshops that principals could access; some partnered with 
universities (as in Charlotte-Mecklenburg’s multi-year induction program) or other providers of profes- 
sional development. 

The districts tried to find ways to customize their professional development offerings for principals, with 
evaluation results as one possible source of guidance. For example, a Denver official described the intention 
as “shifting from sort of ‘one size fits all’ to a more differentiated approach.” A first step in that direction 
was to offer more choice in professional development offerings, but a district leader explained in 2014 that 
the next step would be to move beyond relying on a voluntary approach and instead determine “how can 
we guide people” to the resources they needed. As of 2015 no district claimed to have successfully com- 
pleted this work. 


Accomplishments , Challenges, and Lessons 

Most principals saw evaluation systems as fair and constructive 

Over three years during which the districts changed their systems of principal evaluation, novice principals 
gave these systems consistently positive reviews on our surveys. Their responses differ from those reported 
in earlier research which found that, on the whole, principals saw limited usefulness of performance evalua- 
tion for professional learning (Portin, Feldman, & Knapp, 2006). Most novice principals who had been 
evaluated in 2013-14, in every district, agreed that the evaluation system was accurate and fair, provided a 
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common language for professional practice, set clear and consistent expectations for their performance, 
and was generally useful for purposes of informing their professional practice and identifying areas in 
which they needed to improve, according to our survey (Exhibit 11). Principals responded to a five-point 
scale, and the percentage of respondents giving one of the top two responses is shown in the exhibit, by 
year. (The other three responses were “somewhat,” “minimally,” and “not at all.”) On an additional ques- 
tion asked only in 2015, 57 percent of responding principals agreed to a great or considerable extent that 
the previous year’s evaluation “provided results that were worth the effort.” 


Exhibit 11: Principal perceptions of their 2012, 2013, 
and 2014 performance evaluation 


The performance evaluation system: 

Percent of principals responding “to a considerable extent” 
or “to a great extent,” by evaluation year 

2012 evaluation 

(N= 204 ) 

2013 evaluation 

(N=259) 

2014 evaluation 

(N= 30 o) 

a. Was useful for purposes of informing my 
professional practice 

65% 

54% 

60% 

b. Provided clear expectations for my 
performance that were consistently and 
clearly communicated 

64 

58 

66 

c. Adequately/accurately reflected my 
performance 

63 

54 

60 

d. Was useful for purposes of 

identifying/defining my professional 
development needs 

58 

50 

56 

e. Captured the breadth and complexity of my 
leadership role 

58 

49 

58 


Exhibit reads: In 2013, 65 percent of second- and third-year principals in the six districts reported that their 2012 performance evaluation was useful for purposes of informing their 
professional practice to a considerable extent or to a great extent. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 


The evaluation systems gave principals suggestions for improvement. More than half of novice principals 
were told after their 2013-14 evaluations that they needed to improve in at least one area of practice, ac- 
cording to surveys administered in 2015. This was the case for 69 percent of principals who had been in 
their first year as principals in 2013-14 and 59 percent of those who had been in their second year. 
Instructional leadership was the area in which they most commonly reported having been told they needed 
to improve. This is likely to mean both that novice principals had more to learn about instructional leader- 
ship and also that principal supervisors were paying particular attention to this area of leadership practice. 




Among those novice principals who were given any areas of improvement after a 2013-14 evaluation, 
about half reported on the survey that they had been told they needed to improve in at least one of the fol- 
lowing practices, which we classified as instructional leadership: 

Helping teachers improve their professional practice 
Pressing teachers to set high standards for student learning 
Using data to inform instruction 
Tracking student academic progress 
Conducting instructional planning with teams of teachers 

The largest percentages reported needing to improve in interaction with teachers, including helping teachers 
improve their professional practice and pressing teachers to set high standards for student learning. 

Novice principals participating in focus groups expressed general approval of their districts’ evaluation sys- 
tems. Some described particular strengths of these systems, including the regular visits from their supervi- 
sors and the feedback conferences that were built into the evaluation procedures. A principal in Gwinnett 
County, echoing the intentions expressed by many district leaders across the six districts, explained that he 
did not think of “this big evaluation” but 

...an ongoing conversation all the time about what are your goals, how are you working toward 
those goals, and are you making progress or not. So it’s not sit down and have one meeting and be 
evaluated with feedback for next year because it’s an all-the-time conversation. 


Evaluation rubrics were still a work in progress in 2015 

Some principals participating in focus groups offered suggestions for improving the evaluation rubrics used 
to measure their professional practice. Principal supervisors and district staff were working on this. The 
rubrics often included examples of principal practice to illustrate evidence of effectiveness, but some did not 
explicitly define the amount and type of data, by standard, that a supervisor would need to assemble in 
order to evaluate principals accurately. For example, one rubric offered the wording, “Helps teachers and 
staff set, monitor, and achieve challenging goals based on student outcomes; anticipates, adapts, and per- 
sists in the face of obstacles and responds in a positive solutions-oriented manner,” — but it did not specify 
the type of data a supervisor would use to measure this competency. 

District leaders wrestled with tensions between the value of accommodating differences among principals 
and schools and the value of holding uniformly high expectations. One principal in a focus group voiced 
this common dilemma when he criticized the system that held his work in a turnaround school to the same 
standards of performance as principals who were in higher performing schools and principals who had 
been in their schools longer: 
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... so coming in as a new principal this year with no [grace period], the walls are really high [but] 
you’re getting this “Go slow to go fast; build relationships” message and you’re like, “Great, that 
doesn’t perfectly align to the framework of how I’m evaluated.” 

Supervisors in several districts believed that their district’s evaluation rubric should differentiate expecta- 
tions by principal experience or school context, although none of the rubrics in use in 2015 did so. District 
leaders wrestled with the issue of differentiation, acknowledging that these concerns were legitimate and 
reasonable. Still, they chose to hold constant the goal of effective leadership in every school. A district 
leader in Hillsborough County, although aware of the problems, argued that the expectations for all princi- 
pals had to be the same because “the outcomes for students that we’re expecting are the same.” 

Debate also emerged about whether rubrics should include measures of school practice — that is, the condi- 
tions in the school. One administrator in New York City explained that a heavy emphasis on measures of 
school practice had become a way to “unearth” problems in the school — such as struggling teachers or a 
negative school culture — that should be a principal’s responsibility to solve. “You do not hear people try to 
skirt responsibility for the [school] piece [anymore].” On the other hand, a supervisor explained that she 
thought this was a flawed strategy because it crowded out measures of some principal skills: “It doesn’t re- 
ally get to the dispositions, the ability to communicate, the ability to strategize.” 


Principal supervisors were seen as helpful, although their skills were still developing 

Principals in several districts described a sea change in the frequency and quality of supervisor visits to their 
schools. Having their supervisors in their schools more regularly was a welcome and, in their view, effective 
policy change. A principal described the supervisor visits in 2014-15 as having an entirely different shape, 
rhythm, and intensity than in past years: 

...this year, my [supervisor] is seeing me in the midst of the work where he’s attending our leader- 
ship team meetings, he’s at our faculty meetings, he’s in our grade-level planning meetings and 
sharing ways that I might [do things differently], and just really helped focus on what’s the data, 
what’s our story, what are we doing well and what are we not doing so well and how can we refine 
those practices. 

Tension between the principal supervisor’s dual role of evaluator and support provider did not appear to be 
a serious problem. Some principals said in focus groups that they were cautious about disclosing their trou- 
bles to their supervisors. In general, however, the survey data for 2015 suggested that novice principals 
viewed supervisor support favorably. While coaches were still the preferred source of support as they had 
been in the previous year, supervisors were starting to close the gap (Exhibit 12). In 2013-14, there was a 
statistically significant difference between principal perceptions of the support they received from their 
mentor or coach and perceptions of the support they received from their supervisor/ evaluator. By 2014-15, 
the differences had narrowed so much that they were no longer statistically significant. 



Exhibit 12: Difference in principal perceptions of the support they received 
from their supervisor/evaluator and mentor/coach in 2013-14 versus 2014-15 
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Exhibit reads: In 201 3-14, 73 percent of novice principals who agreed that their mentor/coach had helped them select professional development that meets their needs compared with 58 percent 
who agreed that their supervisors supported them in this way, which was a difference of 1 5 percentage points. In 2014-1 5, the difference between principals who agreed that their mentor/coach 
versus their supervisor/evaluator helped them select professional development that met their needs had dropped to six percentage points. 
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Focus group comments suggested that supervisor support could be inconsistent, however. Principals men- 
tioned supervisors who were difficult to work with, were unwilling or lacked the capacity to help a princi- 
pal address a need, or simply did not show up in the school often enough to help the principal develop 
professionally. Concerns about supervisors’ administrative and central-office obligations came up in each 
district. One principal said: “...if their job is to be instructional coaches to us as school leaders and 35 per- 
cent of their time is taken up by meetings in this [district headquarters] building, how are they coaching us 
and how are they helping us?” 

For some supervisors, shifting into the role of principal supporter and coach was a challenge. Leaders in 
every district said that the shift away from focusing on operations management made the journey difficult 
for some supervisors. Some also pointed to the challenge facing novice supervisors — and for the district in 
guiding them: 

Our [principal supervisors] are still young in their leadership in this capacity, and again, it’s about 
teaching adults. They are not the principal anymore. They are the supporter of principals and that’s 
a different lens and we are still learning what that looks like. 


Individual support was perceived favorably, but professional development less so 

Principals appreciated their supervisors, as just described, and they expressed particular enthusiasm about 
their mentors and coaches. Both of these structures for individual support developed over the years of the 
initiative. Not only did the districts have time to build arrangements and staff skills, but individual school 
leaders had years of working with assistance providers. A coach traced the influence of the preparation 
pipeline in describing the depth of training and support principals receive: 

We’re getting principals that have had support since the time they were assistant principals, so 
that’s a lot of just one-on-one conversations. ... the value of that is just amazing. Because they can 
ask us anything, and not that we know the answer, but we keep asking them questions and helping 
them look at their school. 

Principals’ comments in focus groups painted a picture of the coach or mentor as their lifeline, providing 
day-to-day, hands-on support that principals said was vital to their immediate survival but that their super- 
visors usually did not have the bandwidth to provide. Principals described their mentors as “cheerleaders,” 
“someone to bounce ideas off of,” “available at all times,” “an amazing resource,” “the most effective 
source of support,” and a “lifesaver.” One principal’s description reflected the perspectives expressed by 
many across the districts: 

Flaving a [mentor] has probably been the most effective [support]. And I know for my mentor, it’s 
a 24-hour-a-day job. I mean, I’ve had to have those middle of the night conversations with my 
mentor about things that keep me up at night, and she’s just been very open and very wise and 
given great advice. So that has probably been the number one [thing] for me as a new principal is 
having that [mentor]. 



Having the support of a mentor or coach appeared to raise principals’ opinions of the effectiveness of all 
the support they received from their district. Principals who had mentors or coaches were more likely to 
rate the support they received from their supervisor and from professional development higher than princi- 
pals who did not have a mentor or coach in 2015. While the differences in the average ratings of support 
were not large — means of 4.2 versus 3.8 for supervisor support and 4.0 versus 3.8 for professional develop- 
ment — they were statistically significant. Indeed, the differences suggest that districts’ investments in men- 
tors or coaches extended the benefits of all the support principals received. 

Systems to link mentoring or coaching with professional development or with principals’ individual evalua- 
tion results were not fully developed as of 2015. According to the survey responses, it appears that princi- 
pals and assistant principals struggled to find relevant professional development and pursue their own 
plans for professional growth. 

Although pipeline districts were working to strengthen the alignment, they continued to offer traditional 
courses and classes that principals in focus groups described as “compliance-oriented,” intended for mass 
consumption, and overloaded with information. No principal referred to district-sponsored professional de- 
velopment as helpful or focused on addressing individual needs. As one district leader explained, even when 
professional development courses are phenomenal, you still need individualized support to improve as a 
school leader: “At the end of the day, humans need other humans to help them get better at this work.” 

The relatively low ratings for traditionally organized professional development suggest that districts might 
consider scaling back their investment in it. Reasonable questions have emerged about whether current dis- 
trict-run professional development systems are worth shoring up and reshaping. What would it take, for 
example, for districts to create a system that directly, practically, and quickly responds to principals’ needs? 
A district leader in Denver described the complexities of building such a system: “That’s just challenging, to 
take a principal out of their building, when so much is going on, to offer PD that is just-in-time, that’s dif- 
ferentiated, that is really meaningful, and actually moves the growth of that leader. ” 


Summary 

Districts implemented standards-based systems for evaluating principals’ professional practice, and novice 
principals received critical feedback from those systems. Their perceptions of the systems remained positive 
over the years. The districts’ efforts to shift the role of principal supervisors to emphasize support with in- 
structional leadership paid off in rising principal ratings of the help they received from supervisors, who 
were not only their evaluators but also, for some principals, partners in an “all-the-time conversation” 
about professional practice. 

Principals generally expressed appreciation for the one-on-one support they received from mentors, coaches, 
and supervisors. Group-based professional development was less highly rated, and district leaders contin- 
ued to wonder how — or whether — to continue trying to use that mode of support for on-the-job learning. 
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6. The Assistant 
Principalship: Dilemmas 
of the Position 


O ther chapters of this report have focused on 
problems that district leaders perceived in 
2011 and have described the strategies that the 
Principal Pipeline Initiative offered for addressing those 
problems during the next few years. The story is a bit 
different for the assistant principalship: if anything, the 
problems with that position came into increasingly clear 
focus over the years of the Principal Pipeline Initiative. 
District leaders recognized that substantial work lay 
ahead if they were going to bring coherence to the roles 
of assistant principals within their leadership pipelines, 
but they remained unsure of what to do. 


Takeaways 

1 . The assistant principalship had two 
functions that were hard for districts to 
reconcile: the position was a training ground 
for principals but also the place to assign 
managerial tasks. 

2. Districts addressed the pipeline function 
of the assistant principalship by taking steps 
toward systematically selecting, inducting, 
and coaching assistant principals. 

3. Assistant principals' opportunities to 
practice instructional leadership on the job 
remained somewhat limited. 


An Unsolved Problem: How to Position the Role in a Pipeline 

High proportions of the districts’ novice principals came from the ranks of assistant principals. This was 
true of more than two-thirds of the novice principals responding to our survey each year in each district. 
The proportion was more than 90 percent in Gwinnett and Hillsborough Counties, where there was a pol- 
icy of relying heavily on an in-district pathway to principalship through the job of assistant principal. On 
the other hand, because all of the districts had more assistant principals than principals, many assistant 
principals were not destined to advance. 

District strategies for working with assistant principals treated them as apprentice principals in some ways. 
By 2015, districts had moved to offer more advancement opportunities to selected assistant principals. But 
this collided with two realities: much of the work of the assistant principalship was not about instructional 
leadership; and many of the people in that job were not likely to become principals. 

In a way, districts started with a solution: they hoped to use the assistant principalship strategically, as both 
apprenticeship and proving ground for future principals. Most of the assistant principals continued to see it 
this way as well: among first-, second-, and third-year assistant principals responding to our surveys in 
2013, 2014 and 2015, over four out of five either planned to apply for principal positions or already had 
done so. In Charlotte-Mecklenburg and Hillsborough County the proportion was over nine out of ten. 
Assistant principals reported in the survey that they were attracted to the principalship by the responsibility 
for student achievement gains and the prospect of influencing school change. The professional status of 
being a principal motivated relatively few assistant principals, according to their survey responses. 
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Seeing that they were not necessarily offering assistant principals a solid apprenticeship in instructional 
leadership and that the selection process for advancement was leaving many assistant principals unsatisfied, 
district leaders continued to worry about the assistant principalship in 2015. Some wondered whether they 
should restructure the position. None claimed that they had good ideas about how to address the problems. 


Strategies: Trying to Adapt Pipeline Components for Assistant Principals 

Basing standards and evaluation measures on those developed for principals 

New York City was the only one of the districts that had not built out standards for assistant principals in the 
same way that they had for principals. An interviewee commented on this fact in 2014, implying that it was 
an area for future improvement: “An interesting thing in our system is that there’s a very rigorous and well- 
defined set of standards and a process for evaluating both teachers and principals, but not so much for APs.” 

Interviewees in the other five districts reported that standards for assistant principals were the same as those 
for principals, but that a principal’s level of proficiency or performance was not expected of assistant princi- 
pals. “Those exact same competencies and beliefs are now being used with our APs, because we expect the 
same skills, competencies, and beliefs,” said a Charlotte-Mecklenburg official — but she then qualified the 
statement with, “Possibly to a different level for a first-year AP.” Other districts had spelled out some differ- 
ences in expectations. Prince George’s County standards for assistant principals closely resembled those for 
principals, but the indicators were expressed in terms of “supporting” the principal in carrying out functions 
like the school budget, which officially belonged to the principal. A Denver interviewee said that the stated 
leader standards were the same for both roles, but added, “the evidence guides are differentiated, so what’s 
‘effective’ at the AP level would look slightly different than what’s ‘effective’ at the principal level.” 


Preparing educators to become building leaders 

In their work with preparation programs, the districts sought to improve future principals’ skills in instruc- 
tional leadership. All districts formed partnerships with universities for program improvement. This work 
changed the preparation of assistant principals, since programs of principal preparation were typically pre- 
requisites for certification as an assistant principal. Hillsborough County also started an in-district prepara- 
tion program for aspiring assistant principals in 2011, and Gwinnett County added more formal structure 
to its existing program. 


Offering programs and coaching for novice assistant principals 

Four districts built induction support for new assistant principals. Hillsborough County launched a two- 
year induction program for assistant principals under the initiative, focused on individual mentoring. Each 



novice assistant principal was assigned a mentor who would supplement the informal mentoring that prin- 
cipals were expected to offer to their own new assistant principals. A cadre of “master mentors” helped to 
oversee the mentoring. Gwinnett County also introduced mentoring for assistant principals under the initia- 
tive. Survey results from assistant principals showed that these districts stood out from others in 2015 for 
the higher rates of individual mentoring (Exhibit 13). 


Exhibit 13: Percentage of novice assistant principals 
who reported having another mentor or coach 
besides their principal, by district, by year 
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Exhibit reads: The percentage of assistant principals reporting having another mentor or coach besides their principal was 40 percent in 2013, 26 percent in 2014, and 23 percent in 
2015 in Charlotte-Mecklenburg. 

* A significantly higher percentage reported having a mentor or coach in GCPS and HCPS than in the other districts. 

**201 3 NYC assistant principal data were not included in analyses due to a low response rate. 

Source: Assistant Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015. 
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Prince George’s County added individualized professional development for assistant principals through a 
new induction program. A formal program of support for new assistant principals also started up in 
Charlotte-Mecklenburg. That two-year program included monthly meetings for professional development 
led by district staff on topics such as instructional coaching and school improvement plans. The district also 
offered an Assistant Principal Academy at a nearby university designed to help the assistant principals see 
their opportunities for influence as school leaders. A district official expressed the hope in 2014 that this 
work would bring results: 
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I think that we’ve recognized that we could do a better job of helping with the induction of our 
new APs. Because that’s our real, best, true pipeline to the principalship, so if we spend a lot more 
time grooming and developing them it will pay off in the end. 

New York City recognized learning opportunities for assistant principals as a possible area for improve- 
ment. An official in New York City said: 

There are programs for assistant principals that are designed to help them become principals but 
not [programs that say,] you just became an AP and you want to learn how to do the job well and 
you need some support. We don’t have strong centrally based programs for that. 

Some efforts were under way to ensure that high-potential assistant principals would not be neglected after 
their initial induction into the position. Charlotte-Mecklenburg created a central-office position in leader- 
ship development in 2014, with a mission of building skills among both assistant principals and principals. 
Denver started a program called AP Ascent in the same year, seeking to bolster assistant principals’ skills at 
a time when they were a few years into the position but not yet on the verge of applying for principalships. 
A cohort of participants had group sessions together while each took on a leadership project in his or her 
school. In Hillsborough County, an administrator described the aim of building principals’ skills in talent 
development — but expressed some uncertainty about what principals were doing: 

We did a blended coaching training. ...to get our principals thinking differently about working 
with their APs. We also started the creation of learning plans. ... In the summer, [principals] ... set 
goals with their APs and they’re supposed to support their growth. That’s where I lose track of, 
okay, is that happening? So ... we are going to do more with principals that have these aspiring 
leaders.... Because what we want to do is build the capacity of the entire system to improve their 
coaching and mentoring skills. 


Accomplishments , Challenges , and Lessons 

Some reported that preparation and support for assistant principals were effective 

A top official in Hillsborough County offered an assessment in 2015 of changes made in school-leader 
preparation that was typical of the positive comments made across districts: 

We are now seeing candidates that are coming to the table for both the assistant principal and the 
principal role that are so ready. I mean, they’re ready to move into that role, the rigor that they 
have gone through, ... the way those programs have now been revitalized, so to speak, or initiated, 
has really created a right skill set for everyone coming to the table. So that’s huge. 

In Prince George’s County, mentor principals spoke in 2015 of the ways in which their developmental sup- 
port for new assistant principals had changed. 



[It feels] a little different now than it has in the past.... And so they’re now tailoring professional 
development and differentiating by those individuals that just arrived as assistant principals. And I 
haven’t seen that type of development in the past before. And I think they’re very appreciative of 
it... If we want to change the perception of what an assistant principal is, the only way to do that is 
to change the way that we interact with them. And so that work is new this year also. 

In focus groups, new and more experienced assistant principals spoke highly of formal preparation pro- 
grams and induction programs. Just as important, they repeatedly claimed that the best preparation they 
received was working day-to-day, in schools, with a principal. 


Many assistant principals did not have the benefit of systematic evaluation procedures 

Evaluation procedures and rigor were said to vary within districts. The instructions for evaluation generally 
consisted of variants on principal evaluation systems, but interviewees reported that there were differences 
in execution within the districts and even within large high schools with multiple assistant principals. One 
superintendent described the evaluation system as “building the building.” The superintendent continued, 
describing the evaluation system for assistant principals: 

I think we’re doing evaluations. They’re evaluated on the same instrument as principals. Their 
competencies are the same. We have district wide competencies around leadership, I should say.... 
[The principals conduct the evaluations.] It really is the quality of the feedback and is as strong as 
the principal in terms of how seriously they take their role as a developer of people. 

The official standards for assistant principals could pose challenges for principals who tried to apply them 
faithfully in evaluation. By mirroring the standards for principals, the standards could be a poor fit for the 
job of assistant principal. A principal in a focus group explained some of those challenges, saying that the 
use of principal standards with assistant principals was: 

...confusing both for an assistant principal ... [and] for me as a principal. If I’m evaluating my assis- 
tant principal and [the standards are] saying she should be doing all the things I’m doing, I don’t think 
that [she should be]. ... She doesn’t get to set a new vision ... because I’m running the school and I 
have a vision. And as an assistant principal, your job is to get on board with some of those things. 


Many assistant principals lacked opportunities to build or maintain instructional 
leadership skills 


An official in one district spoke frankly about how the position of assistant principal might allow skills to 
atrophy: 


I think where we have fallen short is during that time that they’re in that assistant principal’s role. 
We have not capitalized on it being a developmental [opportunity]. ...They take the position and it 
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gets very, very narrow. They’re doing discipline only.... The role of the assistant principal has got 
to be redefined, because we’re losing capacity when they’re in the role. It’s actually taking them a 
step back in most cases. 

Another district official gave a similarly bleak assessment: “I would say it’s one of the most neglected positions 
in many districts in terms of professional development, really thinking about where they fit into your pipeline.” 


Districts had to work with the many assistant principals who would not advance 

Because not all assistant principals showed high potential for the principalship, districts faced the challenge 
of moving some of their ambitious assistant principals off the track to a hoped-for promotion. Changes in 
principal hiring procedures, such as multiple selection stages, had the effect of narrowing the field of poten- 
tial applicants, but there were still many assistant principals who cherished the hope of advancing. 

Gwinnett County leaders were comfortable with the idea that some of their sitting assistant principals were 
likely to spend their careers in that position, and in that district a sizable group of assistant principals 
(about 30 percent) responded on the survey that they did not intend to apply for principalships. In other 
districts, however, leaders expressed concern about the disappointment that lay ahead for current assistant 
principals who had not shown high potential to meet the district’s expectations as principals. These leaders 
were beginning to strategize about career opportunities other than principalships that might appeal to these 
assistant principals, such as central-office positions. Such exit strategies for sitting assistant principals could 
free up their positions for new assistant principals who would bring not only ambition but also skills in in- 
structional leadership. 

Complicating this kind of planning, however, was the recognition that districts also needed a cadre of career 
managers to help run school operations, particularly in large schools. Differentiating instructional leadership 
tasks from operational management within the assistant principal role became a central dilemma facing the 
districts as they worked to more formally integrate the role of assistant principal into the leadership pipeline. 


Summary 

While the core focus of the districts in the Principal Pipeline Initiative was the development of principals, dis- 
trict leaders came to realize the importance of bringing coherence to the position of assistant principal as 
well as the promise and challenges associated with finding an appropriate place for that position within the 
pipeline. How to bring coherence was not clear, however. Lightly adapting the tools of standards and evalua- 
tion for assistant principals was not necessarily helpful for the assistant principals. Looking ahead, districts 
had to balance the need to develop and manage staff to help run schools with the need to leverage the role of 
assistant principal as a stepping stone to the principalship. To complete the work that districts have begun in 
thinking about and reengineering the role of the assistant principal, they will likely need to follow a similar 
path to the one they have thus far taken in making system-level improvements in their principal pipelines. 



7. Conclusions and 

Implications 

■ 

r he Principal Pipeline Initiative challenged a 

group of districts to push beyond the work that 
they were already doing on school leadership by 
designing and carrying out system-level improvements 
based on a general template of key pipeline compo- 
nents. Those components were: school-leader standards 
and competencies; selective principal preparation pro- 
grams reflecting district priorities and research-based 
features; selective, data-informed hiring and placement; 
and standards-based evaluation accompanied by more 
intensive support for principals. 

Results of the Initiative 

THE PRINCIPAL PIPELINE INITIATIVE SUCCEEDED IN DRIVING THE INTENDED CHANGES IN 
POLICY AND PRACTICE. This is a surprising finding because large urban districts do not necessarily respond 
to outside initiatives with serious, sustained implementation efforts. These districts, however, did just that. 


Takeaways 

1 . All six districts carried out the kinds of 
changes that the initiative called for. 

2. Changes in standards and hiring gave the 
initiative a strong start. 

3. More new principals over time said they 
were an "excellent" fit in their schools, and 
district leaders saw greater strengths in their 
instructional leadership. 

4. Building capacity for principal preparation 
and support took years and was still a work 
in progress. 

5. The initiative offers implications for both 
districts and policymakers. 


Changes were visible in each component 

ALL SIX DISTRICTS MARSHALED THE WILL AND CAPACITY TO PUT THE KEY COMPONENTS 
IN PLACE BY SPRING 2015: 

They had — and actively used — standards for principals. 

New and revamped preparation programs were running. 

Hiring and selection procedures incorporated more data and longer-term planning. 

New systems of principal evaluation and support were maturing. 


Districts had the will and skill to keep improving their pipelines 

THE DISTRICTS EMBRACED KEY TECHNOLOGICAL AND STAFFING SUPPORTS for the major 
components of the principal pipeline: Leader Tracking Systems brought data together to inform hiring and 
other choices; and principal supervisors took on larger roles in succession planning, principal evaluation, 
and support for new principals. 
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MORE BROADLY, DISTRICT LEADERS REFLECTED ON THE PATHWAYS AND LEARNING OP- 
PORTUNITIES THAT THEY OFFERED to educators at career stages beginning before an assistant prin- 
cipalship and culminating in the role of principal supervisor. By 2015 they had a clearer view of their 
overall systems of school leadership, appreciating how the components of these systems interacted to shape 
professional progress over time for individuals and the quality of school leadership for the district. And 
they were poised to continue taking stock of principal preparation, hiring, evaluation, and support, and to 
continue iteratively refining these system components. 


Early results of system changes were evident among novice principals 

DISTRICTS WERE DOING BETTER OVER TIME IN DEVELOPING AND PLACING NEW PRINCIPALS, 

according to our survey data from those principals. More novice principals over time reported an “excel- 
lent” fit between their own capacity and their school placement. Their reports also pointed to a preparation 
experience that, overall, increasingly fostered their orientation to the principalship as a career and a hiring 
process that gave them a greater chance to show their skills in practical simulations. Top district leaders re- 
ported in interviews in 2014 and 2015 that incoming principals and assistant principals were showing lead- 
ership strengths, especially in instructional leadership. The principals’ own increasing reports of an 
“excellent” fit between their skills and their schools’ needs were consistent with these reports from district 
leaders. 

NEW SYSTEMS OF EVALUATION AND SUPPORT DREW GOOD RATINGS from the new principals 
affected by those systems. Those who had mentors or coaches continued to rate that source of assistance 
positively. Support from principal supervisors garnered higher ratings from novice principals over time. 

And despite the introduction of new evaluation practices and measures, which might have caused anxiety 
or unhappiness, new principals’ ratings of district evaluation systems held steady rather than dropping. 

OVERALL, THEN, WE FOUND THAT THE DISTRICTS SUCCEEDED IN FOLLOWING THE GUID- 
ANCE OF THE PRINCIPAL PIPELINE INITIATIVE; that they built tools, programs, partnerships, data 
systems, and staff capabilities for supporting novice principals; and that early evidence from principals 
showed signs of positive results from the work that the districts did. Exhibit 14 summarizes the districts’ 
strategies, challenges, and successes, by component. 



Exhibit 14: Summary of implementation strategies 
and reported results through 2015 


Principal standards and competencies 

Strategies 

Worked on principal standards at the start of the initiative, incorporating mandated state 
standards and engaging stakeholders, including principals, in deliberations. 

Continually revisited and revised standards based on pipeline implementation experiences. 

Used the standards and competencies to align components of the pipeline, including 
building preparation curricula, hiring criteria, and tools for principal evaluation and 
support. 

Challenges 

Superintendent transition could affect the stability of standards. 

Successes 

Districts reported standards gave valuable structure, clarity, and coherence to district 
policies on principal evaluation, hiring, and preparation. 

Some described the standards as providing a helpful shared language about the 
principalship. 

Principal preparation 

Strategies 

Launched new preservice programs, including in-district programs and programs co- 
developed with partners. 

Worked to strengthen programs through systematic review and communication with 
partners. 

Developed strategies and culture for identification and selective recruitment of aspiring 
leaders. 

Worked to improve and expand clinical experience in preservice programs. 

Challenges 

Changing preservice preparation is a multi-year strategy for improving principal capacity. 

Providing high-quality on-the-job experiences can be costly and relies on mentoring 
capacity and skill that may be limited. 

Successes 

Districts believed the portfolios of preparation programs were stronger because they: 

• Used a selective admissions process. 

• Permitted district leaders to conduct classes and personally observe and 
evaluate rising talent. 

• Tailored program content to district leadership standards, routines, and priorities. 
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Exhibit 14: Continued 


Principal hiring and placement 


Strategies 

Engaged top-level district leaders in succession planning and placement discussions. 

Introduced systematic data-informed procedures for hiring, including performance 
assessments such as role playing and school data review. 

Built new individual-level data systems to track experience, performance, and assessed 
competencies of principals and aspiring principals. 

Incorporated additional selection stages, including hiring pools and preservice program 
selection and performance. 

Challenges 

Using performance assessments in the hiring process and developing and using individual- 
level data systems require significant investments of staff time. 

Selecting principal candidates with strong interpersonal skills is hard, because such skills 
are difficult to measure during the hiring process. 

Successes 

Novice principals reported that the hiring process gave them useful feedback. 

Over time, more novice principals reported an "excellent" fit between the needs of their 
schools and their skills, experiences, and interests. 

District leaders applauded novice principals' strong knowledge of instruction. 

Evaluation and support of novice principals 

Strategies 

Developed an evaluation system that was aligned with principal standards and promoted 
ongoing, regular discussion between principals and their supervisors. 

Expanded corps of principal supervisors and shifted their role toward supporting 
principals in instructional leadership. 

Provided support to novice principals through mentors or coaches. 

Challenges 

In evaluation, uniform expectations were in tension with the differences among principals 
in years of experience and school context. 

Shifting supervisors away from management and compliance roles and toward skilled 
principal coaching and support was a work in progress. 

Reshaping professional development to align with principals' individual needs had limited 
success. 

Successes 

Novice principals reported that the evaluation system was accurate and useful for 
informing their practice and described their supervisors, mentors, and coaches as 
important sources of support. 







What Helped the Districts Make Progress 


District leaders maintained a focus on the key components of the Principal Pipeline Initiative and made 
changes that were consistent with those components as originally designed. This was the case despite super- 
intendent transitions in four of the districts (Charlotte-Mecklenburg, Hillsborough County, New York City, 
and Prince George’s County) and despite the many other initiatives under way in all six districts. How did 
this happen? 

Our interviews and observations over time suggest that THREE FEATURES OF THE INITIATIVE DE- 
SIGN MADE A DIFFERENCE. FIRST, THE DESIGN HAD BOTH “TIGHT” AND “LOOSE” ELE- 
MENTS. The components were defined, and districts were accountable for taking action on each one. 
However, there was “looseness” in the latitude offered for different tactics and emphases. For example, 
partnerships with universities could be a centerpiece of principal preparation, as in Charlotte-Mecklenburg, 
or a complement to in-district preparation, as in Gwinnett and Hillsborough Counties. They could even 
change direction after a few years, as in New York City’s 2014 shift from an effort to influence curriculum 
to a design for changing the clinical experience within city schools. 

SECOND, THE INITIATIVE WAS DESIGNED TO ENGAGE TOP DISTRICT LEADERS in matters that 
otherwise might have remained routine parts of the work of human-capital departments. Superintendents 
were part of the ongoing conversation with the foundation and attended the initiative’s Professional 
Learning Community meetings. This appeared to ensure durable high-level support for the initiative within 
districts. 

THIRD, THE INITIATIVE GAVE THE SIX PROJECT DIRECTORS REGULAR OPPORTUNITIES TO 
MEET TOGETHER, WORK WITH OUTSIDE EXPERTS, AND VISIT EACH OTHER’S DISTRICTS. 

The Professional Learning Community meetings allowed them to share ideas with each other and also to 
spend uninterrupted discussion and planning time with key members of their own local leadership team, in- 
cluding the superintendent. The project directors said their work had benefited from all these opportunities 
for interaction. 

SUPPORTED BY THESE DESIGN FEATURES, THE DISTRICTS TOOK CHARGE OF THEIR OWN 
WORK. Local commitment to the initiative did not erode but actually appeared to us to strengthen over the 
years. Superintendents actively promoted changes that cut across the purviews of different district divisions, 
like principal supervisors’ new responsibilities, and this in turn reduced the chances that bureaucratic miscom- 
munication or resistance could impede the changes. The local project directors remained passionate advocates 
of the initiative in their districts, and they deepened their knowledge about challenges and opportunities in the 
work. They approached their tasks not as hoops to jump through for a funder but as part of a coherent de- 
sign. Their creativity, confidence, and skills were important assets in keeping commitment alive. 
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DISTRICTS WORKED ON EACH PIPELINE COMPONENT IN EACH YEAR, striving for continuous 
improvement as flaws came to light in tools or procedures. They streamlined the initial standards docu- 
ments for usability; then revisited their standards and evaluation measures as the use of these tools brought 
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gaps and ambiguities into focus. No district leader characterized any part of the pipeline design as com- 
pleted and unchangeable. Instead, all remained open to improving their work on the basis of experience 
from year to year. 


Persistent Challenges 

THE DISTRICTS STILL HAD WORK TO DO IN DEVELOPING THE CAPACITY TO CARRY OUT 
REDESIGNED WORK in principal preparation, selection, placement, evaluation, and support as of 2015. 
For example, principal supervisors were continuing to learn how to evaluate the practice of new principals 
and coach them effectively for instructional leadership. Moreover, some principal supervisors were still ex- 
pected to carry out administrative tasks, leaving less time than was needed for principal support. 

DISTRICT LEADERS REPORTED THAT THEIR EFFECTIVENESS IN RESHAPING UNIVERSITY 
PROGRAMS HAD BEEN MIXED. Programs and professors already had curricula that satisfied state re- 
quirements, an adequate stream of new enrollees, and a culture that prized academic freedom. They did not 
have to embrace new program designs, and some reportedly did not. It is possible, however, that changes in 
these programs are simply slower to take place than the kinds of changes that a district can influence more 
directly, and that preparation programs were in a process of evolving over time. 

SOME CHALLENGES THAT SURFACED DURING THE INITIATIVE WERE STUBBORN ONES FOR 
WHICH RESEARCH AND PRACTICAL WISDOM OFFERED NO SOLID SOLUTIONS. Districts 
would have liked to screen aspiring principals for qualities that they described as emotional intelligence, mi- 
cropolitical skills, or relationship-building skills. They were not yet confident that they had found good 
ways to do this, although they continued to try. Group-based professional development for new principals 
was not winning high marks from the principals, and the idea of tailoring these conventional professional 
development offerings to individual needs, while appealing, was hard to carry out. The role of assistant 
principal had built-in tensions because it was a stepping stone for some but a dead end for others. 


Implications for Districts 

The experience of these districts offers lessons for other districts interested in doing similar work. Although 
these districts benefited from high capacity and the support provided through the initiative, the following 
steps would be feasible and potentially productive in other large districts, and most of them could be 
adapted for districts of any size: 

The process of defining standards and competencies for principals was a powerful first step. 
Having standards allowed the districts to anchor job descriptions, hiring criteria, and evaluation 
rubrics in their priorities for principals. Broad-based participation in setting the standards gave 
them visibility and credibility. Other districts can now consider starting with the recently released 
Professional Standards for Educational Leaders (National Policy Board for Educational 
Administration, 2015). 



The hiring process was another productive starting point. Districts addressed their pressing con- 
cerns about the caliber of principal candidates by making procedural changes like requiring candi- 
dates to demonstrate their skills, drawing on consultant help to identify performance tasks. 

The districts saw advantages in district-run preparation programs that were specifically tailored to 
district priorities and needs and that gave hiring managers a chance to observe aspiring principals. 

With the Quality Measures tool, they had a structured process for holding preparation programs 
accountable for inputs and outcomes. For a smaller district, simpler versions of these approaches 
could include in-district study groups for aspiring leaders and more regular dialogue with nearby 
preparation programs about district needs and expectations. 

Leader Tracking Systems offered usable information for hiring and succession planning and, over 
the longer term, a chance to improve all parts of the pipeline on the basis of results. Elaborate sys- 
tems would not be needed in the smallest districts, but good records of individual leaders’ accom- 
plishments and career paths would be informative in any district. 

Districts introduced most of their new tools, programs, and procedures on a pilot basis, subject to 
learning and improvement over time. This practice seems to have eased the potential stress around 
principal evaluation systems. For any new pipeline component, a process of piloting and continu- 
ous improvement gave a needed opportunity to spot and correct unanticipated problems. 
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Implications for Policy 

Policymakers who recognize the centrality of the principal in school improvement could offer help for dis- 
tricts in building a stronger corps of novice principals, building on the experience of this initiative: 

These districts found that standards provided a strong start toward an aligned pipeline. Supporters 
of reform could ensure that districts have access to the Professional Standards for Education 
Leaders and examples of job descriptions, evaluation rubrics, and other usable materials based on 
these standards. 

This initiative established four key pipeline components but allowed considerable latitude for local 
invention and initiative. Other agencies that oversee or support work on principal pipelines could 
consider a similar mix of tight and loose coupling. 

Seed money could provide important start-up resources: expert technical help with hiring proce- 
dures or data systems; initial training for those who are taking on new roles in principal prepara- 
tion, evaluation, or support; and opportunities to visit and learn from other districts. 
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The knowledge base on development and support of novice principals has room to grow. Districts 
remained frustrated with the state of the art in identifying high-potential candidates for principal- 
ships, tailoring principal professional development to address identified weaknesses, and helping 
assistant principals along their varied career paths. More work in developing and systematically 
testing new approaches could help. 

Finally, policymakers supporting this kind of major, system-level initiative should recognize that 
putting changes in place takes time, and that the effects of those changes will take still more time to 
emerge. Building a better crop of new principals is a multi-year endeavor that calls for patience. 
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Appendix A: Survey Administration, 
Weighting, and Analyses 


Survey Administration and Response Rates 


In winter 2013, 2014, and 2015, the study team contacted the Principal Pipeline Initiative project directors 
in each of the six districts to explain the purpose of the survey and to request the names and email addresses 
of all first-, second-, and third-year principals and assistant principals. Once we received the principal and 
assistant principal rosters, we reviewed the lists for accuracy. Web-based surveys were administered in spring 
2013, spring 2014, and spring 2015 to all first-, second-, and third-year principals and assistant principals in 
the six districts. We sent multiple reminder emails to non-responding principals and assistant principals. 

Response rates appear in Exhibit A-l. 


In the analysis of survey data, due to the relatively low response rate from principals in New York City (50 
percent in 2015), we tested for non-response bias to determine whether respondents were representative of 
all principals in the district. We used New York City’s progress reports submitted to The Wallace 
Foundation to determine whether there were significant differences between the percentages of survey re- 
spondents who reported having graduated from particular preparation programs versus the percentages of 
graduates of those programs reported in the more complete data from the district. This was the only data 
point in our survey for which frequencies could be obtained from a different and more complete source for 
comparison. The comparison showed that a few programs were overrepresented among survey respondents 
(e.g., three programs were overrepresented in 2015). Therefore, additional analyses tested a sample of sur- 
vey items for differences in responses from graduates of each of those programs versus graduates of all 
other programs. We found only a few significant differences among group means, and these few significant 
differences did not follow a clear pattern or cluster within a particular set of items. We therefore deter- 
mined that within-district survey weights were not necessary for New York City. 


A-1 


Survey Weighting by District 

To determine post-stratification survey weights for each district, the inverse of the number of respondents 
from the district out of the total number of respondents to the survey was divided by six, the total number 
of districts (Exhibit A-2). For example, in 2015, there were 56 principal respondents in Charlotte- 
Mecklenburg and 514 total principal respondents from all six districts, resulting in a survey weight of 1.53 
(i.e., (514 156 ) 16 ). The 2013 and 2014 data from principals and assistant principals were weighted in a simi- 
lar fashion. 
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Exhibit A-1: Survey respondents and response rates, 
by district, 2013 through 2015 


District 

2013 principal survey 

2013 assistant principal survey 

Number of 
respondents 

Response rate, 
in percent 

Number of 
respondents 

Response rate, 
in percent 

Charlotte-Mecklenburg 

43 

96% 

62 

85% 

Denver 

56 

78 

91 

81 

Gwinnett County 

39 

93 

112 

85 

Hillsborough County 

55 

90 

47 

75 

New York City 

102 

31 

N/A 

N/A 

Prince George's County 

58 

89 

36 

78 

Total 

353 

57 

348 

82 


District 

2014 principal survey 

2014 assistant principal survey 

Number of 
respondents 

Response rate, 
in percent 

Number of 
respondents 

Response rate, 
in percent 

Charlotte-Mecklenburg 

52 

96% 

89 

84% 

Denver 

68 

83 

91 

84 

Gwinnett County 

47 

100 

146 

90 

Hillsborough County 

42 

89 

97 

82 

New York City 

269 

51 

208 

54 

Prince George's County 

63 

93 

40 

87 

Total 

541 

66 

671 

73 


District 

2015 principal survey 

2015 assistant principal survey 

Number of 
respondents 

Response rate, 
in percent 

Number of 
respondents 

Response rate, 
in percent 

Charlotte-Mecklenburg 

56 

93% 

95 

81% 

Denver 

55 

83 

87 

86 

Gwinnett County 

41 

98 

149 

87 

Hillsborough County 

46 

81 

116 

78 

New York City 

248 

50 

185 

45 

Prince George's County 

68 

96 

56 

93 

Total 

514 

65 

688 

68 


Exhibit reads: In 2013, 43 principals completed the survey in Charlotte-Mecklenburg, comprising 96 percent of all first-, second-, and third-year principals surveyed in the district. 
Note: 2013 NYC assistant principal data were not included in analyses due to a low response rate. 




Exhibit A-2: Principal and assistant principal survey 
weights by district and year 


District 

2013 

2014 

2015 

Principal 

weight 

Assistant 

principal 

weight 

Principal 

weight 

Assistant 

principal 

weight 

Principal 

weight 

Assistant 

principal 

weight 

Charlotte-Mecklenburg 

1.37 

1.12 

1.73 

1.26 

1.53 

1.21 

Denver 

1.05 

0.76 

1.33 

1.23 

1.56 

1.32 

Gwinnett County 

1.51 

0.62 

1.92 

0.77 

2.09 

0.77 

Hillsborough County 

1.07 

1.48 

2.15 

1.15 

1.86 

0.99 

New York City 

0.58 

N/A 

0.34 

0.54 

0.35 

0.62 

Prince George's County 

1.01 

1.93 

1.43 

2.80 

1.26 

2.05 


Exhibit reads: The survey weight for Charlotte-Mecklenburg principals in 2013 was 1.37; the survey weight for Charlotte-Mecklenburg assistant principals in 201 3 was 1.12. 

Note: Survey weights for 201 3 assistant principals were calculated out of five rather than six districts. The response rate in New York City among assistant principals was low, so 
those assistant principal responses were excluded from analysis in 2013. 


Survey Analyses 


This report presents some analyses of cross-sectional data from 2013 through 2015. The 2013 data in- 
cluded principals and assistant principals who started in their position from 2010 to 2013, the 2014 data 
included principals and assistant principals who started their position from 2011 to 2014, and the 2015 
data included principals and assistant principals who started in their position from 2012 to 2015. 


We also had the opportunity to go beyond analyzing cross-sectional data and designate cohorts of princi- 
pals who started their position at five different time points. The first three cohorts of principals and assis- 
tant principals were surveyed in 2013, and one new cohort of first-year principals and assistant principals 
was surveyed in each of the two subsequent years for a total of five mutually exclusive cohorts. 


The month of March was used as a cut point to assign principals and assistant principals into appropriate 
cohorts, both because the survey was administered in spring and because a principal appointed near the 
end of the school year could be best considered to belong to the following year’s cohort of new principals. 
We primarily used principal and assistant principal self-reports of their start date, with district survey ad- 
ministration lists as backup, to assign principals and assistant principals to cohorts. See Exhibit A-3 for a 
detailed explanation of cohort assignments for principals and assistant principals each survey year. 
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Exhibit A-3: Principal and assistant principal cohort assignments by year 


2013 

2014 

2015 

Started position from March 2010 
through February 201 1 
(third-year in position) 

Started position from March 2011 
through February 2012* 
(third-year in position) 

Started position from March 2012 
through February 2013* 
(third-year in position) 

Started position from March 2011 
through February 2012* 
(second-year in position) 

Started position from March 2012 
through February 2013* 
(second-year in position) 

Started position from March 2013 
through February 2014* 
(second-year in position) 

Started position from March 2012 
through February 2013* 
(first-year in position) 

Started position from March 2013 
through February 2014* 
(first-year in position) 

Started position from March 2014 
through February 2015 
(first-year in position) 


Exhibit reads: In 2013, principals and assistant principals started their positon from March 2010 through February 2013. For survey analyses, principals and assistant principals 
who started their position from March 2012 through February 2013 were considered first-year principals or assistant principals in 2013; principals and assistant principals who 
started their position from March 2011 through February 2012 were considered second-year principals or assistant principals in 2013; and principals and assistant principals who 
started their position from March 2010 through February 2011 were considered third-year principals or assistant principals in 2013. 

Note: The cohorts that include an asterisk were surveyed in multiple years. 


A-4 


These cohorts served two analytical purposes: (1) to define who was considered a first-, second-, or third- 
year principal or assistant principal each survey year, and (2) to compare principals and assistant principals 
who started the job early in the pipeline initiative to those who started later. 


We used the latter method for survey analyses pertaining to respondents’ reports of past experiences (for 
example, features and perceptions of preservice training and hiring) or factual information that would not 
change over time (such as background information). For these questions, we used the data from the first 
survey year when each respondent had the opportunity to respond to a question. For questions asked in all 
three survey years, we were thus able to compare the two cohorts who started on the job in 2010-12 to 
those who started in 2013-15. In some cases, where survey questions appeared for the first time in the 2014 
survey, the comparison was between the single cohort that started in 2011-12 and the two cohorts that 
started in 2013-15. 


Some comparisons in the report are presented along with information about whether the difference be- 
tween percentages is statistically significant. One method of determining the statistical significance of the 
difference between two percentages is to compare the confidence intervals of the two percentages. 
Confidence intervals provide information about the accuracy of estimated percentages. If the confidence in- 
tervals for two percentages do not overlap, then the difference is deemed statistically significant. 





Appendix B: An Exploratory Comparative Analysis 


O ur principal survey affords an opportunity for an exploratory comparison of principals’ reports on 
their preparation with the findings of an important prior study (Darling-Hammond et al., 2007) 
which helped inform the design of the Principal Pipeline Initiative by identifying features of exem- 
plary preparation programs. The prior study surveyed two groups of respondents: graduates of the pro- 
grams highlighted as exemplars and a national comparison group of principals. Our survey replicates many 
items about preparation program experiences from that study, allowing a comparison of the results. This 
analysis allows us to see how the preparation of novice principals compared with both groups, and whether 
the comparative findings changed over time. 


Caveats are in order for these comparisons. One relates to the time that has elapsed between the prior and 
current research. A national survey conducted in 2013 through 2015 might well be expected to yield results 
different from those of respondents surveyed in the previous decade. It is likely that the preparation pro- 
grams for principals hired recently in the Principal Pipeline Initiative districts reflected current trends in 
school leadership practice, such as newly intensified expectations for data use. Moreover, they might reflect 
attention to the research on principal preparation that had not yet been reported a decade ago. 


With respect to assessments of program content and instructional approaches, and self-reported perceptions 
of their preparation for specific aspects of leadership, we found that the reports of novice principals in these 
districts were fairly similar to those of graduates of the exemplary programs studied by Darling-Hammond 
and colleagues, and somewhat more favorable than those of that study’s national sample. This was true in 
each year of the survey. 


In comparisons where the effect size exceeds .50, we interpret the difference as at least moderate in size and 
therefore likely to be meaningful. Many of the differences that appear in these exhibits, although statisti- 
cally significant, show small effect sizes; in these cases we do not interpret the apparent differences as being 
meaningful. Thus, for example, principals in our 2015 survey reported much greater emphasis on leader- 
ship for school improvement in their preservice than did Darling-Hammond’s national sample (d=. 80) 
(Exhibit B-l). On the other hand, we do not view the apparent difference in emphasis on leadership for 
school improvement between the principals we surveyed and the graduates of Darling-Hammond’s exem- 
plary programs as meaningful; this is because, although the difference was statistically significant, the effect 
size was small (d=-. 24). 


With those caveats and suggestions for interpretation, we show in Exhibits B-l through B-4 the compar- 
isons between our survey results and those of the prior study. 
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Exhibit B-1: Principal perceptions of preservice content 
compared to Darling-Hammond et al. exemplary programs 
and national comparison, 2013-2015 



Darling-Hammond 

PPI Evaluation 

PPI Evaluation 

PPI Evaluation 


et al. data 

Surveys 

Surveys 

Surveys 

To what extent were the 

reported in 2007 

in 2013 

in 2014 

in 2015 

following qualities/practices 

Exemplary 




true of your formal leadership 

Program 

Principal 

Principal 

Principal 

preparation training? 

(N= 242 ) 

( 11 = 326 ) 

(n=472) 

( 1 = 448 ) 

a. Program content emphasized 

4.58 

4.27** 

4.43** 

4.25** 

instructional leadership 

(d=--37) 

'o' 

<N 

II 

(d=--39) 

b. Program content emphasized 

4.49 

4.27** 

4.38** 

4.29** 

leadership for school improvement 

(d=-. 26 ) 

(d=-.i 5 ) 

(d=--24) 

c. Program gave me a strong orientation 

4.39 

4.06** 

4.20** 

4.06** 

to the principalship as a career 

(d=--39) 

(d=-. 23 ) 

(d=--37) 



Darling-Hammond 

PPI Evaluation 

PPI Evaluation 

PPI Evaluation 


et al. data 

Surveys 

Surveys 

Surveys 

To what extent were the 

reported in 2007 

in 2013 

in 2014 

in 2015 

following qualities/practices 

National 




true of your formal leadership 

Comparison 

Principal 

Principal 

Principal 

preparation training? 

(N= 629 ) 

( 11 = 326 ) 

( 11 = 472 ) 

(n=448) 

a. Program content emphasized 

4.13 

4.27* 

4.43** 

4.25** 

instructional leadership 

(d=.l 7 ) 

(d=-39) 

(d=.i4) 

b. Program content emphasized 

3.63 

4.27** 

4.38** 

4.29** 

leadership for school improvement 

(d=-77) 

( d=-99 ) 

(d=. 8 o) 

c. Program gave me a strong orientation 

3.73 

4.06** 

4.20** 

4.06** 

to the principalship as a career 

(d=-39) 

(d=-57) 

(d=-37) 


Exhibit reads: Darling-Hammond etal. exemplary programs responses averaged 4.58 and the national comparison responses averaged 4.13 on the extent to which program content 
emphasized instructional leadership. PPI principal responses averaged 4.27 in 2013, 4.43 in 2014 and 4.25 in 2015. Response means in this table are on a five-point scale in 
which 1 equals “not at all” and 5 equals “to a great extent.” 

Note: T-tests were conducted to determine statistical significance; *p<.05 and **p<.01. Effect sizes were calculated using Cohen’s d. We suggest interpreting effect sizes using d=+/- 
.50 as a minimum threshold for interpreting results as having at least a moderate effect. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015; Principal surveys reported in Darling-Hammond, L., LaPointe, M., Meyerson, D., 
Orr. M. T., & Cohen, C. (2007). Preparing school leaders for a changing world: Lessons from exemplary leadership development programs. Stanford, CA: Stanford University, 

Stanford Educational Leadership Institute. 




Exhibit B-2: Principal perceptions of preservice 
coursework compared to Darling-Hammond et al. 
exemplary programs and national comparison, 2013-2015 


To what extent were the 
following qualities/practices 
true of your formal leadership 
preparation training? 

Darling-Hammond 
et al. data 
reported in 2007 

PPI Evaluation 
Surveys 
in 2013 

PPI Evaluation 
Surveys 
in 2014 

PPI Evaluation 
Surveys 
in 2015 

Exemplary 

Program 

(N= 242 ) 

Principal 

(n= 326 ) 

Principal 

(n=472) 

Principal 

(n=448) 

a. Analysis and discussion of 

case studies 

4.39 

3.96** 

(d=-- 38 ) 

4.15** 

(d=-. 25 ) 

3.86** 

(d=--43) 

b. A portfolio demonstrating your 
learning and accomplishments 

4.36 

3.88** 

(d=s 8 ) 

3.98** 

(d=--3i) 

3.60** 

(d=--54) 

c. Analysis and discussion of field-based 
problems/problem-based learning 
approaches 

4.29 

3.94** 

(d=S0) 

4.12** 

(d=-.i 8 ) 

3.88** 

(d=--34) 

d. Field-based projects in which you 
applied ideas in the field 

4.22 

3.84** 

(d=--3i) 

3.97** 

(d=-. 23 ) 

3.81** 

(d=--33) 

e. Action research or inquiry projects 

4.00 

3.66** 

(d=-. 28 ) 

3.86** 3.61** 

(d=-.i 2 ) (d=-. 3 o) 


To what extent were the 

Darling-Hammond 
et al. data 
reported in 2007 

PPI Evaluation 
Surveys 
in 2013 

PPI Evaluation 
Surveys 
in 2014 

PPI Evaluation 
Surveys 
in 2015 

following qualities/practices 
true of your formal leadership 
preparation training? 

National 

Comparison 

(N= 629 ) 

Principal 

( 11 = 326 ) 

Principal 

(n=472) 

Principal 

(n=448) 

a. Analysis and discussion of 

case studies 

3.74 

3.96** 

(d=.ip) 

4.15** 

(d= 42 ) 

3.86* 

(d=.io) 

b. A portfolio demonstrating your 
learning and accomplishments 

2.81 

3.88** 

(d= 84 ) 

3.98** 

(d=-95) 

3.60** 

(d=-57) 

c. Analysis and discussion of field-based 
problems/problem-based learning 
approaches 

3.47 

3.94** 

(d= 4 i) 

4.12** 

(d=-67) 

3.88** 

(d=-33) 

d. Field-based projects in which you 
applied ideas in the field 

3.37 

3.84** 

(d=-39) 

3.97** 

(d=-55) 

3.81** 

(d=-35) 

e. Action research or inquiry projects 

3.34 

3.66** 

(d= 26 ) 

3.86** 

(d= 46 ) 

3.61** 

(d=. 2 i) 


Exhibit reads: Darling-Hammond et al. exemplary programs responses averaged 4.39 and the national comparison responses averaged 3.74 on the extent analysis and discussion 
of case studies was a part of their coursework. PPI principal responses averaged 3.96 in 2013, 4.15 in 2014 and 3.86 in 2015. Response means in this table are on a five-point 
scale in which 1 equals “not at all” and 5 equals “to a great extent.” 

Note: T-tests were conducted to determine statistical significance; *p<.05 and **p<.01. Effect sizes were calculated using Cohen’s d. We suggest interpreting effect sizes using d=+/- 
.50 as a minimum threshold for interpreting results as having at least a moderate effect. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015; Principal surveys reported in Darling-Hammond, L., LaPointe, M., Meyerson, D., 
Orr. M. T., & Cohen, C. (2007). Preparing school leaders for a changing world: Lessons from exemplary leadership development programs. Stanford, CA: Stanford University, 

Stanford Educational Leadership Institute. 
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Exhibit B-3: Principal perceptions of their preparedness 
compared to Darling-Hammond et al. exemplary programs, 

2013-2015 


To what extent were the 

Darling-Hammond 
et al. data 
reported in 2007 

PPI Evaluation 
Surveys 
in 2013 

PPI Evaluation 
Surveys 
in 2014 

PPI Evaluation 
Surveys 
in 2015 

following qualities/practices 
true of your formal leadership 
preparation training? 

Exemplary 

Program 

(N= 242 ) 

Principal 

( 11 = 326 ) 

Principal 

(n=476) 

Principal 

(i=444) 

a. Engage in self-improvement and 

3.58 

3.38** 

3.36** 

3.35** 

continuous learning 

(d=-. 28 ) 

(d=--3i) 

(d=-. 32 ) 

b. Evaluate teachers and provide 





instructional feedback to support 

3.36 

3.10** 

(d=--3t) 

3.24** 

(d=-.i 6 ) 

3.09** 

(d=-. 3 i) 

their improvement 


c. Create a collaborative learning 

3.34 

3.17** 

3.24** 

3.19** 

organization 

(d=-. 22 ) 

(d=-A2) 

(d=-.i 9 ) 

d. Use data to monitor school progress, 

3.31 

3.18** 

3.16** 

3.08** 

identify problems and propose solutions 

(d=-.i 5 ) 

(d=-.i 8 ) 

(d=-. 26 ) 

e. Design professional development that 

3.25 

2.87** 

2.96** 

2.86** 

builds teachers' knowledge and skills 

(d=--44) 

(d=-. 32 ) 

(d=-.44) 

f. Engage staff in a decision making 


3.13* 

(d=-.u) 

3.11** 

(d=-.i 4 ) 

3.04** 

(d=-. 23 ) 

process about school curriculum and 

3.22 

policies 


g. Lead a well-informed, planned 

3.22 

3.10** 

3.16 

3.05** 

change process for a school 

(d=-.i 5 ) 

(d=-.o 8 ) 

(d=-. 2 i) 

h. Engage in comprehensive planning 

3.22 

3.09** 

3.14* 

3.06** 

for school improvement 

(d=-.i 5 ) 

(d=-. 09 ) 

(d=-.i 8 ) 

i. Create a coherent educational 

3.22 

2.96** 

3.05** 

2.95** 

program across the school 

(d=-- 32 ) 

(d=-. 2 i) 

(d=--33) 

j. Redesign school organizations to 
enhance productive teaching and 

3.06 

2.92** 

(d=-.i 6 ) 

2.93** 

(d=-.i 5 ) 

2.88** 

learning 


(d=-. 2 i) 

k. Understand how different students 





learn and how to teach them 

3.03 

2.70** 

(d=--38) 

2.75** 

(d=-. 32 ) 

2.69** 

(d=-j 6 ) 

successfully 


1. Evaluate curriculum materials for their 

2.92 

2.56** 

2.67** 

2.51** 

usefulness in supporting learning 

(d=--39) 

(d=-. 28 ) 

(d=--44) 


Exhibit reads: Darling-Hammond et al. exemplary programs responses averaged 3.58 on how prepared they were to engage in self-improvement and continuous learning. PPI 
principal responses averaged 3.38 in 2013, 3.36 in 2014 and 3.35 in 201 5. Responses on this table are on a four-point scale in which 1 equals “not at all” and 4 equals “very 
well.” 

Note: Response means from the Darling-Hammond et al. survey were converted from a five-point scale to a four-point scale to allow for comparisons. 

Note: T-tests were conducted to determine statistical significance; *p<.05 and **p<.01. Effect sizes were calculated using Cohen’s d. We suggest interpreting effect sizes using d=+/- 
.50 as a minimum threshold for interpreting results as having at least a moderate effect. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015; Principal surveys reported in Darling-Hammond, L., LaPointe, M., Meyerson, D., 
Orr. M. T., 8i Cohen, C. (2007). Preparing school leaders for a changing world: Lessons from exemplary leadership development programs. Stanford, CA: Stanford University, 

Stanford Educational Leadership Institute. 




Exhibit B-4: Principal perceptions of their preparedness 
compared to Darling-Hammond et al. national comparison, 

2013-2015 


To what extent were the 

Darling-Hammond 
et al. data 
reported in 2007 

PPI Evaluation 
Surveys 
in 2013 

PPI Evaluation 
Surveys 
in 2014 

PPI Evaluation 
Surveys 
in 2015 

following qualities/practices 
true of your formal leadership 
preparation training? 

National 

Comparison 

(N= 62 p) 

Principal 

(n= 326 ) 

Principal 

(n=476) 

Principal 

(n=444) 

a. Engage in self-improvement and 

2.91 

3.38** 

3.36** 

3.35** 

continuous learning 

(d=M) 

(d= 62 ) 

(d=. 6 o) 

b. Evaluate teachers and provide 





instructional feedback to support 

2.82 

3.10** 

(d=-33) 

3.24** 

(d=-54) 

3.09** 

(d= 3 o) 

their improvement 


c. Create a collaborative learning 

2.69 

3.17** 

3.24** 

3.19** 

organization 

(d=- 63 ) 

(d= 7 i) 

(d=. 66 ) 

d. Use data to monitor school progress, 

2.47 

3.18** 

3.16** 

3.08** 

identify problems and propose solutions 

(d=. 82 ) 

(d= 83 ) 

(d=- 70 ) 

e. Design professional development that 

2.50 

2.87** 

2.96** 

2.86** 

builds teachers' knowledge and skills 

(d= 42 ) 

(d= 5 i) 

(d= 4 i) 

f. Engage staff in a decision making 


3 13** 

3 11** 

3 04** 

process about school curriculum and 

2.70 

(d= 52 ) 

(d= 52 ) 

(d= 4 i) 

policies 


g. Lead a well-informed, planned 

2.59 

3.10** 

3.16** 

3.05** 

change process for a school 

(d= 63 ) 

(d=-73) 

(d=- 56 ) 

h. Engage in comprehensive planning 

2.58 

3.09** 

3.14** 

3.06** 

for school improvement 

(d=. 63 ) 

(d=. 68 ) 

(d=-58) 

i. Create a coherent educational 

2.63 

2.96** 

3.05** 

2.95** 

program across the school 

(d= 4 i) 

(d=-53) 

(d= 4 o) 

j. Redesign school organizations to 
enhance productive teaching and 

2.46 

2.92** 

(d=-55) 

2.93** 

(d=-58) 

2.88** 

learning 


(d= 5 i) 

k. Understand how different students 





learn and how to teach them 

2.56 

2.70** 

(d=.i 6 ) 

2.75** 

(d=. 22 ) 

2.69** 

(d=.i4) 

successfully 


1. Evaluate curriculum materials for their 

2.49 

2.56 

2.67** 

2.51 

usefulness in supporting learning 

(d=.o 8 ) 

(d=. 20 ) 

(d=. 03 ) 


Exhibit reads: Darling-Hammond et al. national comparison responses averaged 2.91 on how prepared they were to engage in self-improvement and continuous learning. PPI principal 
responses averaged 3.38 in 2013, 3.36 in 2014 and 3.35 in 2015. Responses on this table are on a four-point scale in which 1 equals “not at all” and 4 equals “very well.” 

Note: Response means from the Darling-Hammond et al. survey were converted from a five-point scale to a four-point scale to allow for comparisons. 

Note: T-tests were conducted to determine statistical significance; *p<.05 and **p<.01. Effect sizes were calculated using Cohen’s d. We suggest interpreting effect sizes using d=+/- 
.50 as a minimum threshold for interpreting results as having at least a moderate effect. 

Source: Principal Survey for “Evaluation of the Principal Pipeline Initiative,” 2013, 2014, and 2015; Principal surveys reported in Darling-Hammond, L., LaPointe, M., Meyerson, D., 
Orr. M. T., 8i Cohen, C. (2007). Preparing school leaders for a changing world: Lessons from exemplary leadership development programs. Stanford, CA: Stanford University, 

Stanford Educational Leadership Institute. 
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